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ABSTRACT 


■  A  syntactic  method  for  the  analysis  of  time-vc.rying  image  potterns 
is  proposed  and  studied.  This  method  utilizes  translation  schema  to 
model  the  time-varying  properties  of  image  patterns.  A  syntactic 
deformation  modec  is  first  applied  to  transform  the  l-th  image  into  the 
(i+1)-th  image  of  an  image  sequence.  Then  the  concept  of  translation  in 
'ormal  language  theory  is  used  as  a  mechanism  to  charac  t  c-r  i  z  e  the 
dynamic  process  of  the  image  sequence.  A  formulation  ot  stochastic 
"ranslation  is  also  presented.  A  generalized  syntax-cn  rect  c-o  tree 
translation  model  is  proposed  to  hanale  high-dimensional  patterns.  The 
generalized  moael  is  compared  with  the  conventional  top-down  ono 
ottom-up  tree  translation  models. 

A  traffic  monitoring  problem  is  analyzed  using  the  proposed  tree 
•ranslation  model.  Each  input  image  is  representeo  as  a  tree  structure, 
he  proposed  tree  translation  model  is  useo  to  model  the  variation  ol 
image  content  between  consecutive  images.  A  parsing  algorithm  for  tree 
■ranslation  is  applied  to  match  moving  objects  (vehicles)  in  each  pair 

uf  consecutive  images,  j 

t . 

I 
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CHAPTER  1 

INTRODUCTION 

Introduction 

During  the  past  two  decades,  there  has  been  an  increasing  interest 
in  pattern  recognition.  Most  of  the  developments  in  the  theory  and 
applications  of  pattern  recognition  use  the  statistical  approach  11-61). 
In  order  to  represent  the  structural  information  containea  in  the 
patterns,  the  syntactic  or  structural  approach  has  been  proposed  L7-10I). 
The  precision  of  syntactic  specification  provides  the  recognition 
procedure  not  only  the  capability  of  classifying  patterns  but  also  the 
capacity  of  describing  patterns.  Recently,  the  problem  of  time-varying 
image  analysis  has  drawn  great  attention  C15-201I.  In  this  research,  this 
problem  will  be  analyzed  using  syntactic  method. 

A  syntactic  pattern  recognition  system  consists  of  3  major  parts: 
(a)  preprocessing  and  segmentation,  (b)  primitive  extraction,  ano  (c) 
syntax  analysis.  To  analyze  time-varying  images,  first  of  all,  we  need 
to  select  a  proper  representation  method.  Then  a  syntax  analysis  scheme 
has  to  be  devised. 

In  the  past,  string  grammar  has  been  applied  to  the  problem  of 
shape  analysis  and  waveform  analysis,  tree  grammar  has  been  applied  in 
the  problem  of  texture  analysis,  fingerprint  recognition,  and  scene 
analysis.  What's  needed  here  is  some  description  method  which  can  be 


applied  to  represent  and  analyze  general  time-varying  image  patterns. 


l~i.  Survey 

1.2.1  Pattern  recognition 

The  many  different  mathematical  techniques  used  to  solve  pattern 
recognition  problems  may  be  grouped  into  two  general  approaches;  namely, 
the  decision-theoretic  approach  and  the  syntactic  approach.  In  the 
decision-theoretic  approach,  a  set  of  characteristic  measurements, 
called  features,  are  extracted  from  the  patterns;  the  recognition  of 
each  pattern  is  usually  made  by  partitioning  the  feature  space  Llj.  Most 
of  the  developments  in  pattern  recognition  research  during  the  past 
fifteen  years  deal  with  the  decision-theoretic  approach  ana  its 
applications  C1-6D.  In  some  pattern  recognition  problems,  the  structural 
information  which  describes  each  pattern  is  important,  ana  the 
recognition  process  includes  not  only  the  capability  of  assigning  the 
pattern  to  a  particular  class,  but  also  the  capacity  to  describe  aspects 
of  the  pattern  that  make  it  ineligible  for  assignment  to  another  class. 
A  typical  example  of  this  class  of  recognition  problem  is  picture 
recognition.  In  this  class  of  recognition  problem,  the  patterns  under 
consideration  are  usually  quite  complex  and  the  number  of  features 
required  is  often  very  large,  which  makes  the  idea  of  describing  a 
complex  pattern  in  terms  of  a  composition  of  simpler  subpatterns  very 
attractive.  In  order  to  represent  the  hierarchical  structural 
information  of  each  pattern,  that  is,  a  pattern  described  in  terms  of 
simpler  subpatterns  and  each  simpler  subpattern  again  described  in  terms 
of  even  simpler  subpatterns,  etc.,  the  syntactic  approach  has  been 


J 


3 


proposed  [7-103.  Iris  approach  draws  an  analogy  between  the  structure  of 
patterns  and  the  syntax  of  languages.  Patterns  are  specified  as  being 
built  up  out  of  subpatterns  in  various  ways  of  composition,  just  as 
ph'-ases  and  sentences  are  built  up  by  concatenating  words,  ano  woras  are 
built  up  by  concatenating  characters.  The  simplest  subpatterns  art- 
called  pattern  primitives.  The  language  that  provides  the  structural 
description  of  patterns  in  terms  of  a  set  of  pattern  primitives  and 
their  composition  operations  is  sometimes  called  the  pattern  description 
language.  The  rules  governing  the  composition  of  primitives  into 
patterns  are  usually  specified  by  the  so-called  grammar  of  the  pattern 
descriptidn  Language. 


A  syntactic  pattern  recognition 

system 

can  be  considered 

dS 

cons i st i ng 

of 

three  major 

parts,  namely. 

preprocess i ng,  pattern 

description 

or 

representat i on. 

and  syntax 

anaLysis  (Figure  1.1). 

T  ne 

f unct ions 

of 

preprocessing 

include 

Ci) 

pattern  encoding 

and 

approximation;  and  (ii)  filtering,  restoration  and  enhancement. 

An  input  pattern  is  first  coded  or  approximatec  by  some  convenient 
form  for  further  processing.  Techniques  of  filtering,  restoration  or 
enhancement  are  used  to  clear  the  noise  and  improve  the  quality  of  the 
coded  patterns.  At  the  output  of  the  preprocessor,  presumably,  we  nave 
patterns  of  reasonably  "good  quality".  Each  preprocessed  pattern  is  then 
represented  by  a  language-like  structure.  This  pattern  representation 
process  consists  of  pattern  segmentation  and  primitive  extraction.  In 
order  to  represent  a  pattern  in  terms  of  its  subpatterns,  we  must 
segmentize  the  pattern  and,  in  the  meantime,identify  the  primitives  in 
it.  In  other  words,  each  preprocessed  pattern  is  segmentized  into 
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subpatterns  an d  oattern  primitives  baseo  on  prespecified  syntactic  or 
composition  operations,  ana,  in  turn,  each  subpattern  is  identified  with 
a  given  set  of  pattern  primitives.  At  this  point,  each  pattern  is 
represented  by  a  set  ef  primitives  with  speeded  syntactic  operations. 
The  decision  whether  or  not  the  represent  a t i on  is  syntact i ca L ly  correct 
will  be  made  by  the  "syntax  analyzer"  or  "parser".  When  performing  tne 
syntax  analysis  cr  parsing,  the  analyzer  can  usually  produce  a  complete 
syntactic  descriotion,  in  terms  of  a  parsing  tree,  of  the  pattern, 
provided  the  latter  is  syntactically  correct. 

For  proper  representation  of  pattern  structures,  different 
languages  has  been  proposed.  String  language  has  been  applied  in  the 
problem  of  shape  analysis  and  waveform  analysis  (Figure  1.2).  Tree 
language  has  been  applied  in  the  problem  of  texture  analysis, 
fingerprint  recognition  and  scene  analysis  (Figure  1.3).  Recently 
attributed  grammar  is  proposed  and  PEE  (  pr imi t i ve-extract ion-embedded  ) 
parser  is  designed  to  increase  classification  performance  £133. 

''.2. 2  Time-varying  image 

The  various  methods  applied  to  the  analysis  of  time-varying  images 
can  be  classified  into  two  groups.  For  research  in  the  first  group 
(cross-correlation  technique  and  image  differencing  technique),  analysis 
operations  are  conducted  at  image  pixel  level  airectly,  while  for  the 
second  group,  actual  analysis  starts  after  some  features  (bounoary, 
etc.)  have  been  extracted  from  each  individual  image  in  the  sequence. 
These  techniques  are  further  described  as  follows: 

rrocs-<-o-roi  of  inn  i-erhniaue  £15,18,22,41,64,65,663:  Leese  et  al. 
C183  qive  a  cyoical  example.  They  compare  two  successive  pictures  with 
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the  first  picture  being  divided  into  systematic  sections  (64x64  pixels). 
Then  for  each  section,  some  reasonable  area  of  the  second  picture  is 
searched  for  a  good  match  to  the  original  section.  They  form  a  cross¬ 
correlation  coefficient  using  the  fast  Fourier  transform  on  the  full 
gray  values  within  the  section.  The  cross-correlat ion  coefficient  is 
computed  for  each  pairing  of  the  originaL  section  with  a  candidate 
section  in  the  second  picture.  The  candidate  section  which  yields  the 
maximum  coefficient  is  chosen  as  the  match.  Then  a  motion  vector  is 
computed  as  the  distance  and  direction  between  the  center  of  the 
original  section  and  the  center  of  the  match  section.  The  motion  vector 
is  essentially  assigned  to  the  section,  not  to  any  object  within  the 
section. 

Image  differencing  technique  123,30,31,46,47,67,68,69,70):  In 
C46,4?D,  the  image  differencing  technique  is  applied  to  find  the 


variation 

between 

two 

images  of  the  same  scene. 

The 

images  are 

carefu  l  ly 

aligned 

by 

both  spatial  coordinates  and 

gray 

value.  The 

spatial  registration  is  done  by  considering  one  image  as  the  reference 
image  and  then  distorting  the  other  image  until  they  are  aligned.  The 
distortion  is  a  localized  procedure  which  operates  on  subregions  of  the 
images.  Cross-correlation  technique  is  used  to  compute  the  amount  of 
distortion  necessary  to  align  a  subregion  with  its  corresponding 
subregion  in  the  reference  image.  After  the  spatial  registration  has 
been  completed,  a  point-to-point  subtraction  process  generates  a  third 
image  which  displays  the  variation  between  the  given  images. 

Limb  and  Murphy  1167]  report  a  hardware  implementation  of 


subtractive  method.  In  addition  to  the  subtraction  process  between  the 
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corresponding  pixels  of  two  consecutive  images,  within-image  comparisons 
are  made  among  pixels  and  their  suitable  neighbors.  The  within-image 
comparisons  are  used  to  normalize  the  between-i mage  comparisons.  These 
comparisons  are  essentially  the  absolute  differences  of  the  pixel  gray 
values  and  are  summed  over  the  entire  image.  This  yielos  a  velocity 
estimate  for  the  image  as  a  whole  and  not  for  any  specific  object. 

The  research  work  in  group  II  includes 
[37,39,42,43,48,49,72,73,74,773:  Chow  and  Aggarwal  l433  develop  an 
algorithm  to  track  moving  objects  in  an  image  sequence.  After 
preprocessing  of  an  image,  they  build  a  model  which  consists  of  the 
objects  extracted  from  the  image.  Each  object  is  represented  as  a  set 
of  feature  values:  area,  centroid  position,  etc.  The  matching  process 
of  moving  objects  between  consecutive  images  is  performed  through  the 
comparison  of  these  feature  values.  After  the  matching,  the  model  is 
then  updated  with  new  information.  The  analyzed  results  include  the 
number  of  moving  objects  and  their  corresponding  velocities  in  the  whole 
sequence.  Other  similar  works  include  [37,39,42,49,773. 

Recently  O'Rourke  and  Badler  report  a  system  for  the  analysis  of 
human  motion  images  [733.  The  model  for  human  body  is  composed  of 
segments  and  joints.  A  joint  is  a  unique  point  connecting  two  segments. 
A  segment  is  an  abstract  rigid  body  with  an  associated  embedded 
coordinate  system.  The  surface  of  each  segment  is  defined  by  a 
collection  of  graphical  primitives,  called  spheres,  located  at  fixed 
positions  within  the  segment's  coordinate  system.  There  are  constraints 
governing  the  relations  between  different  parts  of  the  body.  These 
constraints  arise  from  the  structure  of  the  body,  gravity  effect,  etc. 
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Based  on  the  analysis  of  the  past  images,  the  system  forms  a  motion 
description  which  is  comprised  of  a  set  of  piecewise  linear  functions. 
Each  linear  function  describes  the  location-time  relation  of  a  specific 
feature  point.  The  model  is  updated  with  new  information.  The  system 
then  predicts  searching  regions  of  the  parts  for  the  analysis  of  next 
image.  If  the  prediction  fails  for  a  specific  part,  then  the  system 
terminates  the  analysis  process  for  this  part  and  works  for  others 
hoping  that  the  information  from  succeeding  images  will  resolve  the 
confusion.  The  model  presented  here  is  essentially  an  attributed 
grammar,  but  with  a  slightly  different  form.  Since  it  is  focused  on  a 
specific  object — human  cody,  composed  of  24  segments  and  25  joints,  the 
dynamic  process  of  the  image  sequence  is  represented  as  a  structure- 
preserved  variation  of  attributed  language.  In  other  words,  each  image 
is  represented  by  a  set  of  feature  values  and  the  whole  sequence  is 
represented  by  a  set  of  piecewise  linear  equations.  Besides,  it  is  not 
known  whether  or  not  the  relaxation  process  of  uncertain  part  will 
eventually  be  stable,  which  is  the  common  problem  in  Yachida  et  al.  1773 
and  Tsotsos  et  al.  L7*»3. 

Tsotsos  et  al.  C743  report  a  system  for  the  analysis  of  the  image 
sequence  of  left  ventricular  wall  motion.  They  propose  a  way  to 
represent  motion  concepts  based  on  the  semantic  network  theory.  Each 
motion  concept  is  associated  with  and  defined  by  a  "frame."  A  frame  is 
definable  by  the  user.  Frames  have  an  arbitrary  number  of  "slots”  that 
form  their  parts.  Slots  come  in  two  varieties:  "prerequisite"  and 
"dependent."  Prerequisite  slots  specify  concepts  that  must  be  observed 
before  the  frame  can  be  instantiated,  while  dependents  provide 
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additional  semantic  components  that  are  included  along  with  the  frame 
concept  on  instantiation.  For  example,  the  concept  of  "area  change"  ana 
"contract"  is  defined  as  follows: 

frame  area_change  with 
prerequisites 

subj :  contracti le_object; 
time_int:  time_interval; 
start_a:  area_value; 
end_a:  area_va lue; 

end 

frame  contract  is_a  area_change  with 
prerequisites 

start_a:  such  that 

start_a  >  end_a; 

dependents 

speed:  speed_v  with 

speed  ♦  (start__a-end_a)  i  time_int.  duration 

end 

These  two  frames  illustrate  most  of  the  "syntactical"  construct  of 
their  representation  formalism.  From  the  viewpoint  of  Fu  LIOJ  and  Tsai 
and  Fu  CKJ,  the  representation  method  of  motion  information  describea 
above  is  essentially  that:  each  motion  concept  is  represented  as  a  set 
of  prespecified  attributes,  each  of  which  has  a  special  meaning. 
Tsotsos  et  al.  analyze  the  image  sequence  using  hypotheses-cooperat ion 
method:  uncertain  parts  in  an  image  are  updateu  using  a  relaxation 
process  C883,  during  which  the  iteration  is  not  completed  until  a 
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convergence  to  stable  condition  is  achieved.  It  is  not  sure  whether  or 
not  the  iteration  process  will  always  lead  to  convergence  in  every 
uncertainty  case. 

Yachida,  Ikeda  and  Tsuji  C76]  analyze  the  behavior  of  heart  wall 
motion  by  measuring  the  thickness  of  heart  wall  in  each  image.  The 
thickness  of  the  heart  wall  is  measured  at  a  set  of  boundary  points. 
The  thickness  is  determined  as  the  distance  between  the  points  of  the 
internal  boundary  and  the  points  of  the  external  boundary  intersecting 
with  a  set  of  lines  perpendicular  to  the  internal  boundary.  After  the 
feature  extraction  of  the  whole  image  sequence ,  the  dynamic  process  of 
the  heart  wall  motion  is  represented  as  a  3-D  surface  map,  with  x-axis 
being  the  spatial  domain,  y-axis  being  the  time  domain  and  z-axis  being 
the  thickness.  The  motion  of  heart  wall  is  analyzed  by  observing  the 
surface  configuration.  Heintzen  et  al.  1933  also  represent  the 
information  extracted  from  the  heart  motion  images  as  a  3-D  surface, 
while  Garrison  et  al.  190,91,92,94,953  use  a  curve  (volume-time)  to 
represent  the  dynamic  process. 

The  various  research  areas  of  time-varying  images  are  briefly 
described  as  follows: 

Aerial  and  satellite  images:  Evaluation  of  aerial  and  satellite 
image  sequences  is  an  active  area  in  the  problem  of  time-varying  image 
analysis  Cl  5,1 8,203 .  Wind  velocities  are  estimated  from  cloud 
displacements  observed  in  a  sequence  of  satellite  images  as  regular 
input  data  to  weather  forecasting.  Hubert  et  al.  11213  combine  two  or 
more  copies  from  a  sequence  of  satellite  images  taken  at  20-30  minute 
intervals  over  a  period  of  2-3  hours  into  a  film  loop  which  is 


12 


continuously  projected  onto  a  digitizing  tablet.  An  operator  selects  a 
cloud  feature  (e.g.  cyclonic  vortices,  cold  fronts)  ana  marks  us 
position  in  the  projection  of  initial  and  final  frame  from  the  film  loop 
,  using  the  intermediate  frames  to  securely  track  the  selected  feat  .ires. 
The  measured  displacement  of  the  selected  feature  is  corrected  for 
distortion  and  transformed  to  an  earth  surface  coordinate  system.  On  the 
hypothesis  that  positional  changes  of  cLouds  are  solely  affecteo  by 
horizontal  winds,  the  resulting  cloud  feature  displacement  is  converted 
into  wind  velocity. 

Traffic  monitoring:  Several  investigations  have  been  reported  to 
detect  moving  vehicles  in  sequences  of  film  or  video-films  of  traffic 
scenes  and  to  track  their  motion.  A  reliable  solution  to  this  problem 
would  not  only  allow  to  count  vehicles,  but  in  addition  enable  the 
identification  of  vehicle  type  (car,  truck, bus  ,...)  and  the  observation 
of  vehicle  behavior  for  a  variety  of  quickly  changing  traffic 
situations.  Wolferts  CZ23  described  an  interactive  setup  to  measure 
vehicle  velocity  on  time-lapsed  film  using  cross-correlation  to  tracK 
their  images.  Onoe  et  al.  L2 33  evaluated  video  sequences  for  this 
purpose.  Jones  C24]  reported  on  real-time  tracking  of  features  on 
vehicles  in  video-sequences.  Jain  et  al.  C25J  used  video-sequences  of 
traffic  scenes  to  study  the  detection,  isolation,  tracking,  and 
description  of  moving  objects. 

Industrial  automation:  Application  of  visual  sensors  to  industrial 
automation  appears  to  be  another  area  of  time-varying  image  problems 
C263.  Newmann  C2 7]  studies  the  tracking  of  on  object  on  a  simulatea 
conveyor  belt  by  instantiating  2-D  relational  models  on  straight  line 
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contour  approximations  to  images  of  a  scene  that  contained  a  moving 
object.  If  tracking  has  to  be  performed  under  real-time  constraints,  one 
reverts  to  registering  greyvalue  or  binary  templates  or  to  the 
identification  of  a  feature  that  is  cha'-ac^ristic  for  the  object  in 
question  and  can  be  easily  isolated  within  a  search  window.  Uno  L28J 
employed  such  techniques  on  using  a  TV-camcra  to  detect  the  position  o+ 
bolts  and  reinforcement  ribs  on  moving  steel  moulds  in  order  to  control 
a  manipulator.  Jones  et  al.  C243  investigated  real-time  tracking 
techniques  to  study  the  application  of  fast  visual  feedback  to 
industrial  automation.  Eskenazi  C293  investigated  a  real-time  tracking 
capability  based  on  greyvalue  correlation  by  using  the  video  sensing  and 
processing  setup  developed  for  the  navigable  robot,  by  simultaneously 
tracking  the  same  object  in  two  stereo  image  sequences,  it  is  abLe  to 
determine  the  object's  3-D  trajectory.  Using  this  technique  to  track  a 
stationary  object  from  a  moving  robot,  the  robot's  trajectory  can  be 
determined  in  order  to  guide  its  navigation  and  to  adjust  the  pan  ana 
tilt  of  a  stereo  camera  assembly  on  the  robot  for  keeping  the  reference 
object  in  the  field  of  view. 

Medical  applications:  Medicine  provides  a  major  area  for  image 
sequence  analysis.  The  subtraction  of  X-ray  images  obtained  before  and 
after  injection  of  roentgen-opaque  material  into  the  blood  enhances  the 
resulting  difference  image  for  interpretation  by  a  physician. 
Digitization  of  X-ray  film  images  facilitates  the  performance  of 
nonlinear  operations  such  as  compensation  for  film  characteristics 
during  the  determination  of  the  difference  image.  Although  digitized 
film  images  have  to  be  aligned  prior  to  subtraction,  relative  geometric 
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distortions  can  usually  be  neglected.  Therefore,  the  registration  tends 
to  be  much  less  involved  here  than  with  satellite  images.  If  an  average 
of  several  digitized  pre-injection  images  is  subtracted  from  a  post¬ 
injection  image,  image  components  not  affected  by  the  injection  of 
roentgen-opaque  material  can  be  suppressed  even  more  effectively 
C30,313. 

Sequences  of  X-ray  images  from  the  left  ventricle  have  been 
evaluated  interacti vely  by  cardiologists  to  determine  the  left  ventricle 
volume  or  to  search  for  abnormalities  in  ventricle  wall  motion.  One  of 
the  main  problems  consists  in  a  reliable  determination  of  the  ventricle 
outline  in  the  X-ray  image.  Chow  and  kaneko  L303  searcn  a  suitably 
enhanced  digitized  X-ray  film  image  of  the  left  ventricle  for  subregions 
with  larger  greyvalue  variances.  For  such  subregions,  a  mixture  of  two 
normal  distributions  is  fitted  to  the  corresponding  greyvalue  histogram. 
If  an  acceptable  fit  is  obtained  for  a  subregion  a  threshold  is 
determined  which  is  used  to  classify  the  greyvalues  of  this  subregion  as 
either  interior  or  exterior  to  the  left  ventricle.  In  this  way,  a  series 
of  23  left  ventricular  contours  is  derived  automatically  covering 
approximately  one  cardiac  cycle. 

Injection  of  radioactive  nuclei  into  a  peripheral  vein  and 
subsequent  recording  of  their  decay  gamma  rays  by  a  scientigraphic 
camera  provides  another  way  to  obtain  images  of  the  left  ventricle. 
Although  these  scientigraphic  images  offer  even  less  contrast  than  good 
X-ray  images,  Hachimura  et  al.  1323  were  able  to  determine  the  left 
ventricular  contour  in  such  scient igraphic  image  sequences,  by  detailed 
analysis  of  the  left  ventricular  contour  obtained  at  the  end-diastole 
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and  the  end-systole  they  attempt  to  classify  the  observed  left 
ventricule  wall  motion  into  a  normal  and  an  abnormal  category.  Based  on 
a  time  series  of  nine  sc i ent i grapn i c  images  observed  over  1UU  msec 
intervals  after  the  end-diastole  they  st'^-dte  the  left  ventricular 
volume  and  plot  it  versus  time. 

A  5-0  distribution  of  greyvalues  reflecting  the  spatial 
configuration  of  organs  in  a  living  body  can  be  determined  Dy 
computerized  tomograpny.  A  three  layer  approximat ion  to  such  a 
description  for  the  tnorax  of  a  living  dog  is  given  oy  Johnson  et  al. 
C333.  If  position  ,size  and  shape  of  an  organ  varies  with  time  as  in  tne 
case  of  the  left  ventricle,  such  a  variation  can  be  represented  by  a 
time  series  of  3-D  greyvalue  distributions  which  can  be  considered  as  a 
four-dimensional  greyvalue  distribution.  For  this  purpose,  Herman  ana 
Liu  C  34  J  generalized  a  search  algorithm  by  Liu  L35j  for  the 
determination  of  a  2-0  surface  in  a  3-D  greyvalue  distribution  such  as 
obtained  by  3-0  computerized  tomography. 

Information  about  dynamic  changes  in  size,  shape  and  position  on 
intact  working  organs  is  of  great  interest  not  only  for  the  detection  of 
abnormalities  but  even  for  a  detailed  understanding  of  their  function. 
This  is  not  restricted  to  time  series  of  3-D  greyvalue  distributions  as 
obtained  from  computerized  tomography.  Heintzen  et  al.  L3dJ  emphasize 
this  strong  connection  between  time  sequences  of  single-plane  X-ray 
images  and  a  better  understanding  of  certain  organ  functions. 

Beyond  pure  medical  applications,  the  evaluation  of  image  sequences 
spreads  into  biophysics  and  biology.  For  example,  Yachida  et  al.  L3?j 
observe  fishes  swimming  in  a  vat  by  an  overhead  TV-camera  connected  to  a 
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video-tape  recorder  to  study  their  behavior  unaer  a  variety  of  stimuli 
such  as  lights  or  tones.  Aoout  8  frames  per  second  are  recordea  for 
periods  between  2  and  30  seconds,  resulting  in  sequences  of  20-250 
frames.  Since  the  images  of  these  moving  objects  are  usually  blurred, 
temporal  as  well  as  spatial  greyvalue  differences  are  used  to  separate 
the  images  of  moving  objects  from  those  of  stationary  scene  components. 
Results  from  a  previous  frame  are  used  to  guide  the  feature  extraction 
process  in  the  subsequent  frame.  They  model  the  essential  parts  of  the 
scene  as  it  is  presented  in  each  frame  and  exploit  such  models  obtained 
from  neighboring  -  prior  as  welt  as  posterior  -  frames  to  deduce 
uncertain  parts  or  reanalyze  them.  Davenport  et  al.  L383  also  study  the 
stimulus-response  behavior  of  microorganisms  whose  movement  in  a  wet- 
slide  preparation  under  a  microscope  is  recorded  on  video  tape. 

Ariki  et  al.  L393  design  an  elaborate  interactive  facility  for  the 
analysis  of  image  sequences.  They  analyze  the  morphogenetic  movement  of 
a  dissociated  cell  of  Xenopus  taevis,  a  protozoon.  Operator  intervention 
is  required  to  check  frame  registration  Clow  level  interaction).  Medium 
level  interaction  is  employed  to  define  a  model  for  the  object  that  has 
to  be  traced  throughout  the  image  sequence.  Such  medium  level 
interaction  enable  the  evaluation  of  a  variety  of  real  world  scenes. 
Takagi  et  al.  C403  conduct  a  tracking  problem  in  which  to  determine  the 
paths  of  gramules  in  a  cultivated,  living  pancreatic  cell  recoroeo  by 
time-lapse  cinemicrophotography  at  1  second  intervals  for  5-10  minutes. 
These  granules  carry  insulin  from  ribosomes  where  it  is  produced  to  the 
cell  membrane.  Analyzing  the  motion  of  these  granules  may  therefore 
contribute  to  a  better  understanding  of  hormane  production  mechanism. 
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One  ef  the  problems  in  time-varying  image  analysis  is  the  Drope1- 
representation  of  information  extracted  from  the  image  sequence.  In 
syntactic  pattern  recognition,  a  pattern  is  usually  represented  by  a 
linguistic  notion  called  a  sentence.  The  sent”  ~e  could  be  a  string  ,  a 
tree  or  a  graph  of  pattern  primitives  and  relations.  Most  of  the 
developments  in  syntactic  pattern  recognition  research  during  the  past 
two  decades  deal  with  'static  patterns'  te.g.  fingerprint,  character, 
chromosome  shape).  For  static  patterns,  given  an  input  pattern  x  ana  a 
set  of  pattern  grammars  representing  different  classes,  parsing  scheme 
is  applied  for  the  classification  of  input  pattern.  While  for  time- 
varying  patterns,  after  finding  a  proper  representat ion,  we  need  to  not 
only  analyze  the  pattern  at  each  single  stage,  but  atso  test  the 
mechanism  characterizing  the  dynamic  process  of  the  sequence. 

1  ._3  Summary  of  the  Contents 

Chapter  2  is  concerned  with  the  time-varying  pattern  analysis  using 
a  syntactic  method.  (For  convenience,  we  use  TV  as  an  abbreviation  of 
time-varying.  In  formal  language  theory,  a  translation  is  defined  as  a 
mapping  from  a  language  to  another  language  L^.  If  we  consider  l,,  as 
the  set  of  possible  patterns  occurring  at  time  t^,  \_  as  the  set  of 
oatterns  at  time  t^,  then  the  relation  governing  the  TV  phenomena  could 
oe  formulated  as  a  translation  problem.  In  chapter  2,  we  first  explore 
such  a  translation  using  a  pattern  deformation  model  Cl  1 J  and  tnen 
consider  the  appli cabi L i ty  of  the  translation  models  to  time-varying 
pattern  analysis.  Furthermore,  in  terms  of  the  translation  models,  we 
will  formulate  the  problem  of  TV  pattern  analysis  as  one  which  can  be 
solved  using  traditional  method,  specifically,  context-free  programmed 
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Language  parsing. 

In  chapter  3,  the  concept  of  Language  transLation  is  extended  from 
strings  to  trees  using  a  generaLized  syntax-ai rectea  modeL. 
Conventional  top-down  and  bottom-up  tree  transLation  moaeLs  are  compared 
with  the  generaLized  modeL.  It  is  shown  that  botn  top-down  and  bottom- 
up  models  are  speciaL  cases  of  the  generalized  modeL.  A  parsing 
aLgorithm  for  this  generaLized  tree  transLation  moaeL  is  also  presenteo. 
A  traffic  monitoring  experiment  is  described  in  chapter  4.  Each  image 
of  the  traffic  scene  is  divided  into  a  set  of  windows  which  is  then 
represented  as  a  tree  structure.  A  tree  transLation  schema  is  appiieo 
to  describe  the  motion  of  vehicLes  in  the  input  image  sequence,  the 
matching  process  of  vehicLes  between  consecutive  images  is  performed 
through  a  tree  transLation  parsing. 

Chapter  5  summarizes  the  results  of  this  study  and  proposes 
suggestions  for  further  research. 
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CHAPTER  2 

SYNTAX  OF  TIME-VARYING  PATTERNS 
2_ ._1_  Int  roduct  ion 

In  syntactic  pattern  recognition,  a  pattern  is  usually  representea 
by  a  linguistic  notion  called  a  sentence  C10J.  The  sentence  coula  be  a 
string,  a  tree,  or  a  graph  of  pattern  primitives  and  relations.  Most  of 
the  developments  in  syntactic  pattern  recognition  research  during  the 
past  two  oecades  deal  with  "static  patterns"  (e.g.  chromosome 
c lassi f i cat  ion,  character  recognition,  and  fingerprint  classification). 
For  static  patterns,  given  an  input  pattern  x  and  a  set  of  pattern 
grammars  representing  different  classes,  error-correcting  parsers  are 
used  to  classify  x  into  one  c-f  the  classes  i.10,113.  While  for  TV 
patterns,  given  an  input  seouence  x^,  x^,  ...,  we  need  to  not  only 
analyze  the  pattern  at  each  single  stage,  but  also  test  the  mechanism 
characterizing  the  dynamic  process  of  the  sequence. 

In  formal  language  theory,  a  translation  is  defined  as  a  mapping 
from  a  language  to  another  language  L^.  If  we  consider  as  the  set 
of  possible  patterns  occurring  at  time  t^,  l_2  as  the  set  of  patterns  at 
time  t 2,  then  the  relation  governing  the  TV  phenomena  could  be 
formulated  as  a  translation  problem.  In  this  chapter,  we  first  explore 
such  a  translation  using  a  pattern  deformation  model  Lllj  and  then 
consider  the  app li cabi l i ty  of  the  translation  models  to  time-varying 
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pattern  analysis.  Furthermore,  in  terms  of  the  translation  moaeLs,  we 
will  formulate  the  problem  of  TV  pattern  analysis  as  one  which  can  oe 
solved  using  traditional  method,  specifically,  context-free  programmea 
language  parsing. 

2._2  Preliminaries 

Definitions  and  notations  that  will  be  referred  to  in  tnis  chapter 
are  summarized  as  follows  C 51  — 5A 3 . 

Definition  2.U  Suppose  that  £  is  an  input  alphabet  ana  A  is  an  output 

•jf 

alphabet.  We  define  a  translation  from  a  language  C  £  to  a 
language  9  4  as  a  relation  T  from  £  to  A  such  that  the  domain  of 
T  is  L1  and  the  range  of  T  is  . 

Definition  2.2:  A  syntax-directed  translation  schema  (SOTS  for  short)  is 
a  5-tuple  T  =  (N,£,  A,  R,  S),  where 


(1)  N  is  a  finite  set  of  nonterminal  symbols, 

(2)  E  is  a  finite  input  alphabet, 

(3)  A  is  a  finite  output  alphabet. 


(4) 

R  is  a 

finite 

set  of 

rules 

of 

the  form  A  a. 

6,  where 

a  e  (N 

uD*/ 

8  e  (N 

U  A)*, 

and 

the 

nonterminals  in 

6  are  a 

permutation  of  the  nonterminals  in  a, 

(5)  S  is  a  distinguished  nonterminal  in  N,  the  start  symbol. 

Let  A  *  a,  6  be  a  rule.  To  each  nonterminal  of  a  there  is  an  associated 
identical  nonterminal  of  8.  If  a  nonterminal  B  appears  only  once  in  a 
and  8,  then  the  association  is  obvious.  If  B  appears  more  tnan  once,  we 
use  integer  superscripts  to  indicate  the  association.  This  association 


21 


is  an  intimate 

part  of 

the  rule.  For  example. 

in  the 

ru  le 

A  -*■  B  CB  , 

b(2)b(1)c. 

the  three  positions  in 

, (1 ),c(2) 
b  CB 

are 

associated  with 

positions  2, 

3,  and  1,  respectively,  in 

b(2)b(1)c. 

The 

translation  defined  by  T,  oenoted  x(T),  is  the  set  of  pairs 

* 

£(x,y)|(S,S)  ===>  (x,y),  x  e  ar)d  yet}. 

Definition  2._3:  If  T  =  (N,£,  A,  R,  S)  is  an  SDTS,  then  t(T)  is  called  a 
syntax-directed  translation  (SDT).  The  grammar  G.  =  (N, £,  p r  S) ,  where 

P  =  -CA  +  a  |  A  -»  a,  6  is  in  R>, 

is  called  the  input  grammar  of  the  SDTS  T.  The  grammar 

I  t 

G  =  (N,  A,  P  ,  S),  where  P  =  {A  -  B  |A  -  a,  6  is  in  R>  is  called  the 
output  grammar  of  T. 

Definition  ,2.4:  An  SDTS  T  =  (N,£,  A,  R,  S)  such  that  in  each  rule 
A  •»  a,  8  in  R,  associated  nonterminals  occur  in  the  same  order  in  a  and 
B  is  called  a  simple  SDTS.  The  translation  defined  by  a  simple  SDTS  is 
called  a  simple  SDT. 

Definition  2^.5 >:  A  simple  SDTS  T  =  (N,  A,  R,  S)  such  that  each  rule 

in  R  is  either  of  the  form  A  ♦  a B,  aB  or  of  the  form  A  a, a  where 
A,  B  e  N,  a  c  £  and  a  e  A*,  is  called  a  regular  SDTS.  The  translation 
defined  by  a  regular  SOTS  is  called  a  regular  SDT, 
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Definition  2.6:  For  two  strings,  x,  y  c  53*/  we  define  a  transformation 
T:  ♦  53*  such  that  Y  e  T(x>.  The  following  three  transformations 

are  introduced  C 54 J : 

(1)  substitution  error  transformat  ion 

TS 

o^aw^  ^ - f°r  a'  k  c  23/  a  *  b. 


(2)  deletion  error  transformation 

td 

I —  u)iw2'  ^or  a  e  23  / 

(3)  insertion  error  transformation 

ri 

“lu2  ^ - for  alt  a  e  53  where  ^  e  53 


Example  2.1:  Given  a  sentence  x  =  cbabdbb  and  a  sentence  y  =  cbbabbob, 
then 

TS  TS  TI 

x  =  cbabdbb  | -  cbabbbb  | -  cbabbdb  | -  cbbabbdb  =  y 


Definition  2.7:  A  context-free  programmed  grammar  (CFPG)  is  a  5-tuple  G 
=  (N, 53,  J,  P,  S)  where 

(1)  N  is  a  finite  set  of  non-terminals, 

(2)  53  is  a  finite  set  of  terminals, 

(3)  S  is  the  start  symbol  in  N, 

(A)  P  is  a  finite  set  of  programmed  productions. 
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(5)  J  is  a  finite  set  of  production  labels. 

Each  production  in  P  consists  of  a  label  reJ,  a  core  production  of 
the  torn  A  ■*  a  where  AeN,  a  e  (N  U  <£)*,  and  a  success  branch 
fielo  and  a  failure  branch  field  each  consisting  of  elements  from 

J.  (This  definition  allows  core  production  to  be  A  ♦  A,  whe*-e 

A  e  N  and  X  is  an  empty  string.) 

A  derivation  or  generation  in  6  proceeds  as  follows:  The  first 

production  is  applied  to  the  start  symbol  S;  therefore,  if  production  r 
is  applied  to  the  current  sentential  form  y  to  rewrite  a  nonterminal  A, 
and  if  y  contains  at  least  one  occurrence  of  A,  then  the  leftmost  A  is 
rewritten  by  the  core  of  production  r  and  the  next  production  label  is 
selected  from  the  success  branch  field  of  r;  if  the  current  sentential 
form  does  not  contain  A,  then  the  core  of  production  r  cannot  be  used 

and  the  next  production  label  is  selected  from  the  failure  branch  field 

of  r;  if  the  applicable  branch  field  is  empty,  the  derivation  halts. 

2_.Z  Formulation  of  TV  Patterns  as  _a  CFPL 
To  recognize  noisy  syntactic  patterns,  various  types  of  error 
transformation  models  were  proposed.  Fung  and  Fu  C52D  proposed  a 
structure-preserved  deformation  model  to  handle  substitution  error  for 
string  patterns.  Lu  and  Fu  C 1 1 3  extended  the  model  to  include 
substitution  error,  insertion  error  and  deletion  error,  which  were 
introduced  in  the  previous  section.  It  is  known  that  any  pattern  x  can 
be  transformed  into  any  other  pattern  y  by  a  sequence  of  error 
transformations.  If  we  analyze  a  TV  pattern  x^,  x^,  ...  xn  from  the 
viewpoint  of  deformation  model,  then  the  TV  phenomena  can  well  be 
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interpreted  through  a  sequence  of  properly  selected  error 

transformations  which  transforms  into  x^,  x^  into  -  ...,  and  x^_^ 

into  x  . 
n 

The  deformation  models  proposed  by  Fung  and  Fu,  and  Lu  and  Fu  can 
be  formulated  in  terms  of  translation  scnema.  Consider  patterns  which 
are  described  by  a  regular  grammar.  To  handle  substitution  error,  <a 
regular  SOTS  TIN,  A,  R,  S)  can  be  constructed,  where  £  =  A  is  the 
primitive  set,  R  is  a  set  of  rules  of  the  form 

A  +  aB,  bB  A,  B  e  N,  a,  b  e  £ 

For  each  rule  A  +  aB  in  the  regular  grammar  and  each  terminal  b  e  £, 

add  a  rule  A  *  aB,  bB  to  R.  To  include  insertion  error  and  deletion 

error,  for  each  rule  A  aB  in  the  regular  grammar,  add  a  rule  A  -*■  aB, 
aB  to  R  for  each  a  in  ^  **•  U-Ce>),  where  K  is  a  parameter 

and  e  is  the  empty  string.  For  patterns  which  are  described  by  a 
context-free  grammar,  a  simple  SDTS  can  be  constructed  to  model  the 
three  types  of  deformation.  First  of  all,  the  grammar  is  transformed 
into  its  Greibach  Normal  Form  C513.  Then  the  construction  procedure  of 
a  simple  SDTS  is  similar  to  that  of  a  regular  translation  schema  except 
that:  For  each  rule  A  ♦  aB  in  the  context-free  grammar,  add  A  ♦  aB,  aB 

for  each  a  eCXZ1  U  ***  u  U  ■Ce>) . 

As  far  as  error  modeling  is  concerned,  Lu  and  Fu's  model  can  handle 
any  number  of  errors  occurred.  For  time-varying  patterns,  we  restrict 
the  number  of  errors  no  larger  than  K.  The  difference  between  Lu  and 
Fu's  model  and  this  translation  model  is  that  Lu  and  Fu's  model  handle 
errors  implicitly  (or  recursively)  while  this  translation  model  must 
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explicitly  specify  the  kind  of  errors  it  intends  to  cover.  For  example, 
in  Lu  and  Fu's  model  E  *  a  and  E  +  bE  recursively  describe  any  number 
of  errors  of  inserting  b  in  front  of  a.  But  in  this  l ,  anslation  model, 
a  translation  rule  A  -*•  as,  aB  has  to  be  included  to  cover  each 
individual  error  a  *  a  even  though  a  has  no  limitation.  The  parameter  k 
here  only  indicates  that  the  number  of  errors  a  ■*  a  is  finite.  In  other 
words,  k  could  be  any  positive  integer  but  k  <  ».  Essentially  we  use 
the  translation  schema  to  implement  the  deformation  model.  To  make  it 
suitable  for  an  n-stage  TV  pattern  analysis  problem  with  sequence  x 
^2 r  •■•/  xn+i/  we  9"ive  the  following  definition. 

Definition  2.8:  An  n-stage  SDTS  is  a  5-tuple 

T  =  (N,  R,  S),  where 

CD  N,£,  A,  S  are  defined  as  those  of  an  SDTS. 

(2)  R  is  a  set  of  rules  of  the  form  A  *  a,  ,  B^,  ...,  B  .  where 
a  e  (N  U  £)* ,  B..  t  (N  U  A)*,  and  the  nonterminals  in  B ^  are  a 
permutation  of  the  nonterminals  in  a. 

The  translation  defined  by  an  n-stage  SDTS  T,  denoted  by  t(T),  is  the 
set 

* 

{(X,  y y,  ***  yn)  I  IS,  S,  S)  ===>(x,  yK  ••*,  yn),  x  c£  ,Y  s  c  A  > 

Thereafter  the  analysis  of  TV  patterns  becomes  a  translation  problem. 

Next  we  investigate  the  "parsing"  of  a  translation  beginning  with  a 
definition: 
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Definition  2.9:  Let  T  be  a  one-stage  SDTS 

★ 

t(T)  =  -CCx,  y)  |  (S,  S)  ===>  (x,  y)>. 

Define  the  input  language 

L.j  as  -Cx  |  (x,  y)  e  x(T)  for  all  y> 
and  the  output  language 

as  <y  j  (x,y)  e  t(T)  for  all  x>. 

The  concatenation  of  and  with  respect  to  T  is  defined  as  the 
set  -Cxy  |  (x,  y)  e  t(T)>,  denoted  by  L^. 

Ll2  C  where  L^'L^  =  Lxy  |  x  e  L^,  y  e 

Theorem  2 .V.  Let  L^  and  L^  be  the  input  language  and  output  Language 
respectively  of  a  regular  SDTS  TIN,  21,  A,  R,  S)  then  L  is  a  CFPL. 
(l,j  is  a  context-free  language.) 

Proof :  We  shall  prove  the  theorem  by  constructing  a  CFPG 
6  such  that  L(G)  =  L^. 

A  CFPG  G(N  ,  £  ,  J,  P,  $)  is  constructed  as  follows: 

Step  1:  n’  =  {A1  |  A  e  N>  U  (A2  |  A  c  N)  U  {S>. 

Step  2:  22  =  22  U  A. 

Step  3:  Since  T  is  a  regular  SDTS,  each  production  in  R  is  of  the  form: 
A  *  aB,  aB  or  A  ♦  a,  a  where  A,  BeN,  a  e  a  e  A*. 
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(1)  Add  S  *  -CK  |  K  is  the  Label  of  production  beginning  with 

S1}  $  to  P. 

(2)  If  A  -*■  aB,  aB  is  a  production  in  R  such  that  A,  B  e  N,  a  c  E 

and  a  e  A* ,  then  add  the  production  to  P: 

2  2  1 
A  *  aB  -CK | K  is  the  Label  of  production  beginning  with  B  >  <t> 

(3)  For  each  input  rule  of  R:  A  -  a8  where  A,  B  e  N  and  a  e  £ , 

1  1 

add  the  production  to  P:  A  ♦  aB  S(u)  $>,  where  S(u)  is  the  set  of 
Labels  of  the  productions  added  in  (2)  corresponding  to  the  translation 
rules  with  input  rule  A  ♦  a8. 

(4)  If  A  4  a,  a  is  a  production  in  R,  where  A  e  N,  a  e  £  and 

*  2 

a  e  A  ,  then  add  the  production  A  ■*  a  $  $  to  P.  , 

(5)  For  each  input  rule  of  R:  A  a  where  A  e  N  ana  a  e  Y1 ,  ado 
the  production  to  P:  A^  -»■  a  S(u)  <t>  where  S(u)  is  the  set  of  labels  of 
the  Droductions  added  in  (4)  corresponding  to  the  translation  rules  with 
input  rule  A  a. 

Then  we  shall  prove  that  =  L(G). 

Let  xy  be  in  L^-,.  Then  by  Definition  2.9  there  exists  a  derivation 
sequence 

(S,  S)  ==>  (a.,A^,  a-1A^) - ===>  (a^2 - a^,  o^a., - a^)  =  (x,y). 

Lie 

,  os.  c  A  and  A.  e  N.  From  the  construction  procedure  of 
CFPG  G,  it  is  known  that:  For  each  derivation  step 


(a 


1a2~‘ "aiAi'  a1  a2~ 


*a^ A • )  ==>  (a,^- 


'“ai+1Ai+V 


alV 


”ai +1 Ai+1 ^ ' 


there  exist  the  productions 
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A1 

l 


ai  +1  Ai+1 


and 


ai+1  Ai+1 


in  F 


with  the  success  field  of  the  first  production  containing  the  label  of 

the  second  production.  Besides  the  success  field  of  the  production 

2  2  11 
A  •*  a.  „  A  .  contains  the  laoet  of  the  production  A.  .  ♦  a.  0  A.  ,  for 
ii+ii+l  i+ii+<:i+c 


1  <  i  <  n. 


Therefore,  there  exists 


derivation  in  G: 


S  ==>  S^2  “>  a-jA-jS  -->  a1A1a1A1 


==>  ala2A2a1A1  > 


a1a2A2a1°2A2 


S=>  a1a2 


a  a  .a.  ’**  «  1  A  T 

n  1  2  n-1  n-1 


• • • • 

==>  a. a-,  a  a.  a-,  a  .a  =  xy.  Thus,  L.-,^  L(6).  Similarly,  from 
12  n  1  2  n-1  n  ic 

the  definition  of  G,  it  can  be  shown  that  L(G) C  i  Therefore, 


L^  =  L(G) . 


Definition  2.10:  Let  T  be  an  n-stage  SDTS,  be  its  input  language,  L^, 
•••  Ln+1  be  its  1st,  2nd,  •**  nth  output  languages  respecti vely. 
Define  the  concatenation  of  ,  L2,  **'  LR+^  with  respect  to  T  as 

{V2  ”*  xn+1  1  (x1'x2'  *’*'  xn+1}  £  T(T)>  denDted  as  L123‘"n+r 


Theorem  2.2:  Let  T  be  an  n-stage  regular  SDTS,  L^ ,  L^  ’**,  Ln+1  be  its 

input  language,  1st,  **•  nth  output  languages  respectively.  Then 

L„ .  is  a  CFPL .  The  proof  is  similar  to  that  of  Theorem  2.1 
12  •**  n+1 

1  ^  i 

except  that  for  the  present  case,  N  =  U  (A  |  A  e  N>  U  -CS>,  and  for 

i=1 

each  production  in  T,  there  are  n+1  corresponding  productions  in  G.  (L1 
is  a  context-free  language.) 
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Theoj-jem  2.3:  Let  T  be  d  simple  SOTS,  be  its  input  ana  output 

Languages  respectively.  Then  is  a  CFPL.  (L^  is  a  context-free 

Language . ) 

Proof:  We  sha.L  construct  a  CFPG  G  such  that  LLG)=L^-  Since  T  is  a 
simple  SOTS,  each  production  in  T  is  of  the  form  A  ■*  a,  ft  where 
A  e  N,  n  c  CNUl')*,  Bt(NUA)*,  a,  6  have  the  same  nonterminals  with  the 
same  permutation.  G  (N  1 ,L'‘  ,J  ,P,S)  is  constructed  as  foLLows: 

Step  1.  N'  =  {A1 I Ar  N>  U  |  AeN)  U  TS>. 

Step  ?.  1''  r  L  U  A. 

Step  3.  (1)  Add  S  +  S^2  <K|K  is  the  Label  of  production  beginning 

with  S1}  a  to  P. 

(2)  For  each  production  A  -*•  a^A^  ...  <*nA  a  .,  S1A1  •** 

★  ★ 

BAB  .  in  R,  where  A.cN,  a.c  Z  ,  BeA  ,  AeN,  ada 
n  n  n+1  '  i  '  i  i 

11  1 

A  *  a. A,  ...  a  A  i< | K  is  the  Label  of  proauction 

11  n  n  n+1 

A  -  61A1  ...  An6n+.|  >  $ 

2  2  2 

and  A  -►  b.A.  SAB  .  <K|<  is  the  label  of  productions 

11  n  n  n+1 

beginning  with  the  leftmost  nonterminal  of  the  present 

derivation  string)-  41 

Strictly  speaking,  there  is  little  restriction  on  the  next 
step  of  a  derivation  after  one  application  of  the  production 
beginning  with  a  nonterminal  of  the  form  A“~  where  A  is  in  N. 
The  only  restriction  is  that  the  next  applied  production  must 
begin  with  a  nonterminal  of  the  form  A1  where  A  is  in  N. 

The  proof  that  L(G)=L^  is  similar  to  that  of  Theorem  2.1. 
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Corollary  2.1 :  Let  T  be  an  n-stage  simple  SDTS,  L  be  its  input 

language,  and  L^,  ...  Ln+^  be  its  1st,  2nd  ...  nth  output  languages 

respectively.  Then  L  .  is  a  CFPl. 

12.. .n+1 

Rosenkrantz  L 53 3  proposed  the  concept  ot  pr  grammed  grammars  os  a 

new  device  for  generating  languages.  It  is  shown  tnat  the  class  of  CFPl 

properly  contains  the  class  of  context-free  Languages  ana  is  properly 

contained  within  the  class  of  context-sensitive  Languages.  Writing  a 

CFP6  is  very  much  like  writing  a  computer  program  ana  is  a  rather 

straightforward  logical  process  compared  with  the  task  of  writing  a 

context-sensi t i ve  grammar.  Swain  and  Fu  L543  further  appLied  CFP6  to 

pattern  recognition  and  proposed  a  stochastic  syntax  analyzer  for  CFPL. 

Lu  and  Fu  C 1 1 3  then  proposed  an  error-correcting  parsing  algorithms  tor 

CFPL.  By  the  previous  discussions,  it  is  known  that  a  class  of  TV 

patterns  can  be  described  as  a  CFPL.  Then  the  analysis  of  TV  patterns 

becomes  the  parsing  of  a  CFPL  which  has  been  well  developed  by  Fu  et  al. 

This  fact  is  to  say  that  some  TV  pattern  analysis  problems  can  be 

thought  of  as  parsing  of  a  CFPL.  In  other  words,  we  can  infer  a  CFPG  to 

interpret  or  to  generate  the  time-varying  process.  A  CFPL  parser  can  be 

used  to  classify  an  input  pattern  sequence.  Actually  the  parsing  task 

can  be  performed  by  directly  building  a  parsing  algorithm  from  the 

translation  schema  instead  of  constructing  a  CFPG.  This  parser  consists 

of  n  interrelated  "sub-parser"  each  of  which  deals  with  the 

corresponding  x^  input  string.  The  basic  idea  is  that  these  n  input 

strings  x  ...,x  need  not  be  concatenated .  However  the  relation  among 
1 ,  n 

these  subparsers  still  has  the  spirit  of  a  CFPL  parser. 
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In  applying  syntactic  methods  to  pattern  recognition,  one 
dimensional  (string)  gi ammars  are  sometimes  inefficient  in  describing 
high-dimensional  patterns.  For  tne  purpose  of  etfectively  Describing 
high-dimensional  patterns,  Fu  and  Bhoraeva  C  5  5  J  introduced  the 
application  of  tree  systems  to  pattern  recognition.  Lu  and  Fu  LlZ'J 
proposed  error-correcting  tree  automata  for  the  recognition  of  noisy 
patterns.  Five  types  of  error  t ransf ormat ion  are  introduced: 

(1)  the  substitution  of  the  label  of  a  node  by  another  terminal  symbol, 

(2)  the  insertion  of  an  extraneous  labeled  node  between  a  node  ana  its 
immediate  predecessor, 

(3)  the  insertion  of  an  extraneous  labeled  node  to  the  left  of  all  the 
immediate  successors  of  a  node, 

(4)  the  insertion  of  an  extraneous  labeled  .iode  to  the  right  of  a  noce, 

(5)  the  deletion  ot  a  node  of  rank  1  or  0. 

The  three  operations  of  insertions  in  rule  (2),  (3)  and  (4)  are 
named  as  stretch,  branch,  and  split,  respectively,  according  to  the 
relative  position  o f  the  inserted  node  to  the  original  tree. 
Apparently,  the  inverse  operation  of  any  type  of  insertion  is  deletion, 
and  the  inverse  of  deletion  operation  is  one  of  the  three  types  of 
insertion. 

The  deformation  from  a  pure  pattern  to  its  noisy  pattern  can  be 
performed  through  a  sequence  of  application  of  these  error 
transformations.  Through  the  use  of  the  deformation  model,  the 
transformat  ion  concept  can  also  be  applied  to  tree  grammar  to  solve  some 
time-varying  pattern  analysis  problems  which  are  not  easy  to  handle  with 
string  grammar  trans lat ions .  After  introducing  a  brief  definition,  we 
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propose  a  tree  grammar  translation  schema.  (For  detailed  description 
about  tree  grammar,  refer  to  L123). 

Oef ini t ion  2.11:  A  tree  grammar  G  =  (V,r,P,S)  over  <£r>  is  expansive 
iff  each  production  in  P  is  of  the  form 

x 

X„  -  /  \  or  X,  -  x  where  xc£. 

u  y  v  ^ 

1  r  Cx) 

and  Xg,  X.j,...,X  ^  are  nonterminal  symbois. 

Definition  2^_1_2:  A  tree  SDTS  T  is  a  4-tuple  (V,r,R,S)  where  V,r, S  nave 
the  same  meaning  as  for  a  tree  grammar,  R  is  a  set  of  productions  of  the 
form:  A  +  a,  B  such  that  a  ano  6  have  the  same  nonterminals,  B  is  tne 
result  of  application  of  a  sequence  of  error  transformations  to  a  ana 
the  input  grammar  is  in  the  expansive  form. 

An  n-stage  tree  SDTS  can  be  formed  with  each  production  being  of 
the  form  A  -*  a,  ,  ^‘"'^n  where  each  Bi  is  the  result  of  some 
deformation  of  a.  The  concept  of  "programmedness"  can  also  be  applied 
to  tree  grammar.  Consequently,  Lu  and  Fu's  tree  language  analysis 
method  can  be  used  to  analyze  some  more  complex  TV  patterns. 

_2.4_  Stochast i  c  Translation 

Definition  2.13:  A  stochastic  syntax-directed  translation  schema  (SSDTS) 
C83D  is  a  5-tuple  T  =  (N,53,A,R,S)  where 
N  =  a  finite  set  of  nonterminals; 

53=  a  finite  input  alphabet; 

A  =  a  finite  output  alphabet; 
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R  =  a  finite  set  of  rules  of  the  form  p:  A  ■»  a,B  for  A  in  N,  a  in 
(N  U  £)*,  6  in  (N  U  A)*,  0  <  p  _<  1 ,  with  the  nonterminals  in  i 
being  a  permutation  of  those  in  g; 

S  in  N  ~  the  starting  symool. 

Each  nonterminal  in  the  a  portion  of  a  rule  has  an  identical,  associated 
one  in  3  (matching  superscripts  are  used  as  needed  for  repeated 
nonterminals) . 

Definition  2.14:  A  schema  T  is: 

(a)  unrestricteo  if  for  each  rule  p:  A  -►  a, 6  the  probability  p  is  not 
conditioned  on  other  rules  or  events. 

(b)  proper  if  for  each  nonterminal  A  the  probadi l i t i es  of  a L l  rules  in 
which  A  is  the  leftside  nonterminal  sum  to  1. 

We  consider  only  unrestricted,  proper  schema.  Translations  from  XI*  to 

★ 

A  are  produced  as  follows. 

Definition  2.15:  A  translation  form  of  T  is  defined  recursively: 

(a)  1:  (S,S)  is  a  form  with  assdciated  S's; 

(b)  if  d:  (oAa,YA<5)  is  a  form  with  the  A's  assdciated  and  p:  A  a, 6 
is  a  rule,  then  pp:  (uiaa,y86)  is  a  form. 

Definition  2.16:  The  stochastic  translation  from  to  A*  produced  by  T 
with  starting  symbol  S  is  the  set 

t(T,S)  =  {(x,y,p)|x  in  X)*/  y  in  A*,p  =  j  p  for  all  translation  forms  p:  (x,y)> 
T  defines  a  function  :  XI  xA*  *  10, ID  such  that 
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n  f  (i ) 

p  <x,y)  =  1  I] 

i =1  j=1  J 

where  there  are  n  distinct  standard  derivations  (e.g.,  leftmost.)  of  the 
translation  form  (x,y)  with  f(i)  steps  in  the  ith  one,  and  p.^(x,y)  is 
the  probability  assigned  to  the  jtn  rule  of  the  ith  derivation. 

Definition  2.17;  A  schema  T  is: 

(a)  consistent  if 


l  £  PT(x,y)  =  1. 


(b)  simple  if  in  each  rule  p:  A  -*•  a, 6  the  nonterminals  in  a  occur  in 
the  same  order  as  their  associates  in  & . 

(c)  regular  if  all  rules  have  the  form  p:  A  +  xB,  y B  or  p:  A  -*•  x,y 

for  A,  B  in  N,  X  in  ]£/  y  in  A*. 

Example  2.2:  The  occurrence  of  substitution  error  in  a  4-bit  binary  code 
transmission  can  be  described  as  follows: 


T  = 

—  P 

CCS, 

A, 

B,C> 

2  c 

1  p 

0 

A, 

0 

A 

2  s 
\  p 

0 

A, 

1 

A 

2  c 
\  P 

1 

A, 

1 

A 

2  s 

~  P 

1 

A, 

0 

A 

2  c 

—  P 

0 

B, 

0 

8 

2  s 

4  p 

0 

B, 

1 

B 

1  B,  1  B 
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A 

B 

B 

B 

B 

C 

c 

c 

c 


2  Ps 

i  pc 
K 
K 

hi 

hi 
i  fs 

K 

K 


1  B,  0  B 
0  C,  0  c 
OC,  1  c 
1  C,  1  c 
i  c,  o  c 
0,  0 
0,  1 
1,  1 
1,  0 


where  is  the  probability  of  correct  transmission  of  each  individual 
bit,  is  the  probability  of  substitution  error  for  each  individual  bit 
and  P£  +  Pc  =  1.  The  translation  pair  (0000,  0010)  is  derived  as 

follows:.. 

1  P 

S  4 — -  0  A,  0  A 

2  pc 

4 — £  00B,  00B 

7  PS 

4 — -  oooc,  001C 

7  p 

- — -  0000,  0010 


The 


probabi l i ty 


of 


this 


deri vat  ion 


i  s 


1  1  1  1  13 

4  P  x-^-P  x  4  P  x  4  P  =  4r  P  p  . 

2c  2c  2s  2c  16  c  s 

There  is  another  way  to  describe  the  probabilistic  property  of  a 
stochastic  SOTS.  Suppose  that  A  is  a  nonterminal  of  a  stochastic  SDTS 
and  there  are  n  translation  rules  with  A  *  a  as  leftside  (or  input  rule 
as  defined  before) : 


A 

A 


3o 


3? 


1  B 
1  B 
0  B 
C  C 
1  C 
1  C 
0  C 
0 
1 
1 
0 


The  Left  part  above  is  the  set  of  rules  for  input  Language  and  each  rule 

is  followed  by  a  set  of  rules  f^r  output  Language.  Th^  input  string 

0000  is  derived  as  follows:  S-OA  -DOB  -  000C  -0000.  The 

1 

probability  of  this  derivation  is  -~r.  Given  this  derivation,  the  output 

'  °p  P  P  P 

string  0010  is  derived  as  follows  S  —  0  A  —  008  —  001C  —  0010.  The 

conditional  probability  of  the  output  string  0010  given  that  0000  is  the 

input  string  is  P^PS.  The  probability  of  the  translation  pair 

<0000,001 0>  is  Pip  • 

16  c  s 

Def ini tion  A  stochastic  programmed  grammar  [103  is  a  5-tuple 

G  =  (VVJ,P,S),  where 

(1)  is  a  finite  set  of  nonterminals; 

(2)  is  a  finite  set  of  terminals; 

(3)  J  is  a  finite  set  of  production  labels; 

(4)  P  is  a  finite  set  of  productions; 
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(5)  S  is  the  start  symbol,  S  e  V... 

N 

Each  production  in  P  is  of  the  form 

(r)  m  -  r>  S(U)P(U)  F(W)PCW). 

The  core  of  the  production  is  "u  *  n,"  where 

"  e  «tUvn)\(»tUvn>* 

and 

n  e  (VTUv/. 

Each  stochastic  production  in  P  has  a  distinct  label,  r  e  J.  U  e  2J  ana 
W  e  2J  are  called  the  success  and  the  failure  "go-to"  fielas, 
respectively.  P(U)  and  P(W)  are  probability  distributions  associated 
with  the  sets  U  and  W,  respectively. 

In  applying  the  stochastic  production  to  an  intermediate  string  t, 
derived  from  S  under  leftmost  interpretation,  if  £  contains  the 
substring  u>,  then  the  leftmost  occurrence  of  u>  is  expanded  into  n  and 
the  next  production  to  be  applied  is  selected  from  the  success  "go-to" 
field  U  according  to  a  probability  distribution  P(U).  If  t  does  not 
contain  w,  then  5  is  not  changed  and  the  next  production  is  selected 

from  the  failure  "go-to"  field  with  a  probability  distribution  P(W). 

r1  r2  rn 

Suppose  that  S  =  oj  — ->  u  — >  ...  — >  «  .  =  x  is  a  aerivation  of 
1  c  n+ 1 

x,  where  r^  denotes  the  label  of  the  production  used  to  directly  derive 
oK+.j  from  UK.  The  probability,  p^(x),  associated  with  the  i-th 
derivation  of  x  is  defined  as  the  product  of  the  conditional 
probabilities  p(r^ Jplr^ | r^ ) . . ,p(rn | rn_^ ) .  The  interpretation  of  the 


4 
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conditional  probability  p(r^|r^_^)  is  the  probability  of  selecting  r^  as 
the  next  production  to  be  applied  from  the  success  "go-to"  field  if  the 
r^-th  production  is  successfully  applied.  Otherwise,  p(r^|r^_^) 
denotes  the  probability  of  selecting  r^  as  the  next  production  to  be 
applied  from  the  failure  "go-to"  field  of  the  r^-th  production.  pCr^) 
is  assumed  to  be  1. 

Definition  2.19:  A  stochastic  programmed  grammar  G  is  said  to  oe  a 
stochastic  context-sensitive,  context-free,  or  regular  programmed 
grammar,  if  G  is  a  context-sensitive,  context-free  or  regular  pg. 

The  stochastic  language  generated  by  a  stochastic  pg  G  is 

*  pi(x) 

L(G)  =  -C(x,p(x))|x  e  V^,  S  - >  x,  for  i  =  1,...,lc 

and 

k 

Z  p .  (x)  =  p(x)>, 
i=1 

and  k  is  the  number  of  all  distinctly  different  derivations  of  x  from  S 
defined  in  G  and  p^(x)  is  the  prr  bility  associated  with  the  i-th 
derivation  of  x. 

The  stochastic  language  L(G)  is  called  a  stochastic  programmed 
language.  G  is  said  to  be  consistent  if  and  only  if 

z  P<*>  =  1. 
xeL(G) 
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Theorem  2.4:  Let  and  be  the  input  Language  and  the  output 

Language,  respecti veLy,  of  a  stochastic  reguLar  transLation  schema 
T(N,]C#A,R,S),  then  is  a  stochastic  context-free  programmed  language 
(SCFPL).  (L^  is  a  context-free  Language.) 

Proof :  We  shall  prove  the  theorem  by  constructing  a  SCFPG  G  such  that 

—I 

L(G)  =  A  SCFPG  G(N’,  2,  ,J,P,S)  is  constructed  as  follows: 

Step  1.  N'  =  -CA1  |AeN>  U  <A2|AeN>  U  ^S>  . 

Step  2.  £  =  H  U  A. 

Step  3.  Since  T  is  a  stochastic  regular  translation  schema,  each 
production  in  R  is  of  the  form 

P:  A  -*■  aB,  aB 


P:  A  -»  a,  a  where  A,BeN,  aeJ^aeA  .  0  <  P  <  1 


Cl)  Put  all  rules  of  R  which  have  the  same  leftside  nonterminal  in 
a  group  within  which  all  rules  with  the  same  input  rule  are  further 
grouped  together  as  follows: 


pH:  A  -  dy  B1 ,  <*11B1 
P12 :  A  ■*  a1B1'  a1281 


P1n(1):  A  ai81'  “inMl8 


ri'  In  (1 )  1 


P21:  A  -  a2B^,  a21B2 


P_  ,  v :  A  *  a  B  ,  a  ,,B 
Tin  (m)  mm'  mnfmt  l 


m  m'  mn(m)  m 
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or 

Pmn(m)‘  A  am'  amn(m) 

where  0  <  P .  .  <  1 , 

-  ij  - 

A,  B.  r  N, 

ai  e  E/ 

* 

a.  .  e  A  , 

1  J 

n(i)  is  the  number  of  rules  having  the 
same  input  rule  A  -*•  a.B.  or 

i  l 

A  -  a., 

and  m  is  the  number  of  different  input  ruLes 
for  nonterminal  A, 

1  <•  i  <  m,  1  <  j  <  n(i). 

(2)  For  each  rule  P..:  A  a.B of  the  group  formed  for  A 

i  j  i  i '  i  j  i 

in  (1),  add  the  rule  to  P: 

Y  A2  -  a.  .  B^  S(U)  P(U)  <t>  1 

where  y  is  the  assigned  label  for  this  new  rule,  S(U)  and  P(U)  are  the 
success  field  and  the  associated  probability  distributions.  The 
construction  of  S(U)  and  P(U)  is  explained  by  the  following  example: 
Suppose  that  B.  =  A  and  the  set  of  rules  in  R  is  grouped  as  shown  in 
(1).  Then  S(U)  =  ly(.l  >,y(2>,.  .  ./yCm)>  and  P(U>  =  ^  ,F>2, . .  .,Pm>  where 
Y (k )  is  the  label  for  the  corresponding  rule  of  the  k-th  input  rule  of 
A,  which  will  be  described  in  (3),  and  P^  =  P^  +  p.^  +  ...  +  P^n^  is 
the  corresponding  probability. 

(3)  For  each  different  input  rule  A  *  a.B.  of  the  group  formed  for 
A  in  (1),  add  the  rule  to  P: 
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r  A1  -  a.B1  SlU)  P(U) 


$  1 


where  y  is  the  assigned  Label  for  the  rule,  SCU)  ana  PCD)  are  tne 
success  field  and  the  associated  probability  distribution  respectively. 
SCU)  =  •Cy(1)/y(2>#...,y(n(i))>,  where  nCi)  is  the  number  of  translation 
rules  with  the  same  input  rule  A  -*■  a.B-  and  ylk)  is  the  Label  of  the 
rule  in  P  corresponding  to  the  k-th  of  these  translation  rules. 
P(U)  =  {P.,,P2,...,Pn(i)>  ur,ere 


P1  =Pi1/(Pi1+Pi2+"‘+Pin(i))' 


Pk  =  Pik/<Pi1+Pi2't'“*+Pin(i))/  1  -  k  -  n(l)* 

(4)  For  each  rule  P^;  A  *  a^,a^  of  the  group  formed  for  A  in 

(1),  add  the  rule  to  P: 


y  A^  ♦  a..  .  1  $  1 

where  y  is  the  label  assigned  to  this  new  rule. 

(5)  For  each  different  input  rule  A  a^  of  the  group  formed  in 
(1),  add  the  rule  to  P: 

y  A1  -  ai  SlU)  PIU)  «|>  1 


where  y  is  the  label  assigned  for  this  new  rule,  SlU)  and  PIU)  are  the 
same  as  defined  in  13). 

16)  Add  the  rule  to  P: 


1  S  >  sh2  SlU)  PIU)  4>  1 


where 

1  is  the  label 

assigned  for  this 

new 

rule. 

SlU)  = 

{yl1),yl2),...,ylm)>  and 

PIU)  -  f P^ ,p2, • ■ • ^P^)/ 

where  m 

i  s  the 

number 

of  different  input  rules 

with  leftside  nonterminal 

S,  y  li ) 

i  s  the 

label 

for  the  corresponding 

rule  in  P  of  the  i-th 

input 

rule. 

A3 


P.  =P . „+P . _+. . ,+P ,  ....  n(i)  and  P.  .  are  as  defined  (1). 
i  ill?  in(i)'  i  j 

Now  we  provea  that  =  LIG). 

Let  (xy,P)  be  in  L  Then  there  exists  a  set  of  different 

derivation  sequences.  Consider  one  derivation  sequence: 


P1  P2  Pn 

(S,S)  —  (a^A^a^A^)  —  (a^a-jA^a^A^)  •  •" —  (a1a2’ *an'a1a?’ *an^  = 

From  the  above  construction  procedure  of  the  SCFPG  G,  it  is  known  that 
for  each  derivation  step 


P. 

i 


<a1a2...a.A.,a1a2...aiAi)  (a^a2...a^+^A^+^,a^a2...a^+^A^+^), 


there  exist  the  production  rules  in  P: 


Y  A*!  -*  a .  . a1  .  S(U)  P(U)  <t>  1 

l  i+1  i+1 


Y  ‘  -  a1+1A^+1  S(U’)  P(U')  0.  1 

where  S(U)  contains  y'  and  the  corresponding  probability  in  P(U)  is 
P./q^  for  some  value  q..  Besides  there  exists  a  production  rule  in  P; 

y  "  A*T  .  a  A*?  S  CU ’  ')  P(U’  ')  <t-  1 

1-1  IT 

where  SCI)11'  contains  y  and  the  corresponding  probability  in  PClT')  is 

V  (pi/qi)  *  qi  =  V 

Therefore  there  exists  a  derivation  in  G" 


.  1  .1.2  q1  J.2  P1/q1  .1  .2  q2 

S  -  S  ,  a1A1S  a1A1a1A1 
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a1a2*"ai  +  1Ai+1a1a2‘ 


*  *  ai  +1  Ai  +1 


4i  +2 


P  „/q„ 
n  n 


a1a2'--ana1a2",an 


xy  . 


q1‘(P1/q1)‘q2*(P2/q2r 


.q^'tP^/q.). 


•  q  •  (P^/q,) 
n  n 


=  P  *p  p  p 

1  V  V  n 


Therefore,  for  each  derivation  of  xy  in  T,  there  exists  a  unique 
corresponding  derivation  with  the  same  probability  in  G.  Besides,  two 
different  derivations  of  xy  in  T  will  not  map  to  the  same  derivation  in 
G.  (In  other  words,  it  is  a  one-to-one  correspondence).  Similarly,  for 
each  derivation  of  xy  in  G,  there  exists  a  unique  corresponding 
derivation  with  same  probability  in  T  (also  a  one-to-one 
correspondence).  Therefore,  there  exists  a  one-to-one  and  onto 
relationship  between  the  set  of  derivations  of  xy  in  T  and  the  set  of 
derivations  of  xy  in  G.  Therefore, 

(xy,P)  e  L12  <==>  (xy,P)  e  L(G)  . 

L12  =  L(G)  . 


Theorem  _2._5:  Let  and  L2  be  the  input  language  ano  the  output 
language,  respectively,  of  a  stochastic  simple  syntax-directed 
translation  schema  SSSOTS  T(N, £,4,R,S),  then  is  a  SCFPL. 


Proof:  We  shall  construct  a  SCFPG  G  such  that  L  (G)  =  L„ .. .  A  SCFPG 

-  I  C 

\ 

G  (N ' ,  Y1  / J ,P,S)  is  constructed  as  follows: 

Step  1.  N*  =  -C A ^  ( A  e  N>  U  CA2|A  e  N>  U  CS> 

Step  2.  -51  II  A. 

Step  3.  Since  T  is  a  SSSDTS,  each  rule  in  T  is  of  the  form 
P:  A  -*■  a,  3 

where  0  _<  P  _<  1,  A  e  N,  as  (NU^)*:Be  (N  U  A)  ,  and  a, 6  have 
the  same  nonterminals  with  the  same  permutation. 

(1)  Put  all  rules  of  R  which  have  the  same  leftside  nonterminal  in 
a  group  within  which  all  rules  with  the  same  input  rule  are  further 
grouped  together.  The  process  is  demonstrated  by  an  example  for 
nonterminal  A  as  follows: 


P  •  A  cx  Q 

r1 1  •  M 


i'  "ii 


p A  ® 


1'  "12 


P1n (1 ) "  A  "  V  61n(1) 


Pmn(m)‘  A  Jm'  ^mnCm) 

where  0  <  P .  .  <  1  a.  e  (N  U  5])*/  6..  c  (N  U  A)*,  a.,  B ■  .  have  the 
—  i J  —  '  i  ij  i  i  j 

same  nonterminals  with  the  same  permutation,  1  <  j  <  nCi),  1  ^  i  _<  m, 
n(i)  is  the  number  of  rules  having  the  same  input  rule  A  +  a  m  is  the 
number  of  different  input  rules  for  nonterminal  A. 


(2)  For  each  rule  P. .:  A  -  a.,B. .  of  the  group  formed  for 

i  j  i '  i  j 


nonterminal 


(1),  where  a.  =  a. .A, ...a.  A  a.,  .... 

i  ill  in  n  i  (n  +  1 ) ' 


8  .  •  =8  .  ,  A. . . .  B  .  A  8  .  ■  ,  , 

11  ill  1  '  I'’  "  ii  (n+1)' 
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a .  e 

1  k 


D  ✓  8ijk  e  A  ,  Ak  e  N,  1  <  k  <  n+1,  n  >  1, 


add  the  rule  to  P: 

,  A2  *  6ij,fl1  —  «ij„An»ii(n,1)  S<U>  P(U>  ♦  ’ 

where  y  is  the  Label  assigned  for  this  new  rule, 
S  (U)  =  {y(1),y(2),...,Y(m)}'r  PCI!)  =  ■CP^  • /P^J,  y(k)  is  tne  label 

for  the  corresponding  rule  of  the  k-th  input  rule  of  nonterminal  , 
P^  =  pki  +p|<2- ■  -+pkn (k ) f  —  k  —  m'  m  the  number  °*  different  input 
rules  of  ,  p^  is  the  probability  of  the  K-th  translation  rule  of  the 
k-th  input  rule  for  k  n(k)  is  the  number  of  translation  rules  of  the 
k-th  input  rule  for  A^. 

(3)  For  each  different  input  rule  A  -*•  a.  of  tne  group  formed  for  A 

in  (1),  where  ai  =  QioAr**0,i  (n-1)Anain'  aik  6  2T  '  Ak  E  N'  0  -  k  -  n/ 


n  >  0, 


add  the  rule  to  P: 


y  A^  a.nA]...a  ,,aV  S(ll)  P(U)  <t>  1 

1 0  1  i  (n-1 )  n  in 


where  y  is  the  label  assigned  for  this  new  rule, 
S(U)  =  ty  (1 ) ,y (2) . y(n (i ) )>,  P(U)  =  ^p-)/P2'*”'Pn(-i  n(i)  is  the 
number  of  translation  rules  with  the  same  input  rule  A  a^,  y(k)  is  the 
label  of  the  corresponding  rule  in  P  of  the  k-th  translation  rule  with 
input  rule  A*v  pk  =  pik  /  <pi1+pi2+-”+Pin(i))'  plk  ^  the 
probability  of  the  k-th  translation  rule,  1  <  k  <  nii). 


(4)  For  each  rule  P..:  A  of  the  group  formed  for  A  in 

1  J  IT) 


(1),  where  a.  e  V  .  e  A  , 

l  •—  '  l  ] 


U7 


add  the  rule  to  P: 

r  A2  -  S..  S(U)  P(U)  $  1 

where  is  the  Labe',  assigned  for  this  new  rv!  • , 

S(U)  =  {T(1,1),Y(1/2),...,r(1,m(1)),Y(2,1)...y(t,fll(t))} 


P(U)  *P1,rP1,2'”,'P1,fn(1)'P2,1'"’'Pt,m(t)>'  where 

t  ’S  the  number  of  nonterminals  in  N,  m(k)  is  the  number  of  aifferent 
input  rules  for  the  k-th  nonterminal,  ylk,!)  is  the  label  of  tne  rule 
added  in  (3)  corresponding  to  the  £-th  input  rule  of  the  K-th 

nonterminal  of  N,  Pk =  pk, *,1+Pk,i,2+- •  -+pk, *,n Ck,U '  n' Ck'U  1  S  the 
number  of  translation  rules  associated  with  the  Jl-th  input  rule  of  the 
k-th  nonterminal,  P,^^,  pk/t#n(k^)  are  their 

corresponding  probabilities. 

(5)  Add  the  rule  to  P: 


S  -  S1 S2 


SCU)  POJ)  $  1 


where  y  is  the  label  assigned  for  this  new  rule. 


S(U)  -  -Cv(1 )  ,r  (2)  , . . .  ,y  (m)>,  P(U)  -  ■CP1  ,P^,  . . .  ,Pm>  ,  where 

m  is  the  number  of  different  input  ru[es  for  nonterminal  S,  y(i)  is  the 

label  for  the  rule  added  in  (3)  corresponding  to  the  i-th  input  rule  of 

S,  P.  =  P .  „+P  .-.+  ••  *+P .  ....  n(i)  is  the  number  of  translation  rules 

i  il  i2  in  (i ) ' 

associated  with  the  i-th  input  rule  of  S,  P . , ,P . . ,,P .  ...  are  the 

il  '2'  'in(i) 

corresponding  probabilities  of  these  translation  rules. 


The  proof  of  l(G)  =  is  similar  to  tnat  of  theorem  2. 4. 


2.5  Illustrative  Examples 


Example  2.3:  The  following  CFPG  6  generates  the  Language 


C  Hi  n  f"l  i  I  ^  ^ 

LSQ=<a  bed  |n>1> 


which  could  be  interpreted  as  the  language  of  squares  of  side  length  n  = 
1,2,...,  CIO!] 


6SQ  =  ^N'VT'J,P'S) 


where  the  vocabulary  consists  of 


VN  =  {S,A,B,C,D> 
a  c 

VT  =  {.*,  bi,  +,  d+> 


the  label  set  is 


J  =  <1,2,3,4,5,6,7>, 


and  the  production  set  P  consists  of  7  rules: 
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Success 

Fai  lure 

Labe  l 

Core 

branch 

branch 

1 

S 

-¥ 

aAB 

{2,3> 

4> 

2 

A 

-¥ 

aAC 

12,3} 

$ 

3 

A 

-*• 

D 

{4} 

9 

4 

C 

- 

o 

{5} 

{*} 

5 

D 

bDc 

£6} 

•y 

6 

B 

-*■ 

d 

{7} 

9 

7 

i 

D 

- 

be 

9 

9 

However,  from  a  different  point  of  view,  this  grammar  can  also 
describe  the  following  TV  pattern:  { (x  ,x£,x^,x^)  |  is  a  line  segment, 
x.  leads  x.  .  bv  9CT,  and  each  x.  has  the  same  length.} 

Example  2.^\  Zucker  £56}  used  the  concept  of  transformational  grammar  to 
analyze  texture.  He  first  built  an  ideal  texture.  Then  a  set  of 
transformation  rules  were  used  to  transform  the  ideal  texture  into  a 
real  texture.  Using  translation  schema,  we  can  nandle  these  problems 
too.  For  example,  consider  the  pattern  shown  in  Fig.  2.1.  A  2-stage 
tree  SDTS  with  substitution  error  only  can  be  used  to  characterize  such 
a  time-varying  image  sequence.  For  an  iteratively  varying  pattern  (like 
TV  texture),  an  n-stage  tree  SDTS  can  be  constructed  to  model  its 
evolving  process. 

Example  2.5:  With  the  development  of  CT  (Computerized  Tomography) 
technique,  a  3-D  object  is  usually  represented  as  a  sequence  of  slices 
(e.g.  X-Y  plane  cross-sections  along  Z-axis).  The  translation  schema 
can  be  applied  to  describe  these  3-D  objects.  Let  Cx^x^,. .  .xn>  be  a 
sequence  of  X-Y  plane  cross-sections  with  Z-coordinate  being  1,2,. ..n, 
an  n-stage  SDTS  can  be  constructed  and  then  transformed  to  a  CFPG.  Here 
a  simple  example  is  given  for  an  object  described  by  4  slices  (Fig. 
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This  object  is  described  by  the  following  "programmed"  SOTS 
T(CS,A,8,C,D,E,F >,£,£,  J,  R,  S) 


where  £=  {  a  /  ,  b  |  ,  c  \  ,  d  -  R  contains 

babet  Core  Success 

Field 


1 

S-ASCDEF 

,ABCDEF 

,ABCDEF 

,ABCDEF 

•C2,3> 

2 

A-*d 

,d 

,d 

/d 

3 

A-»-oA 

,dA 

,dA 

,dA 

C5> 

U 

B-*-a 

,ab 

,ab 

/a 

•C6> 

5 

B»a8 

,aBb 

,aBb 

,aB 

■C7> 

6 

C-*-c 

/C 

,c 

•C8> 

7 

C->cC 

,cC 

,cC 

,cC 

C9> 

8 

D>d 

,d 

/d 

/d 

-Cl  0> 

9 

D-*-dD 

,dD 

,dD 

S 

CL 

•cm 

10 

E+a 

,ab 

,ab 

,a 

C12> 

11 

F-aE 

,aEb 

,aEb 

,aE 

C13> 

12 

F+c 

,c 

/C 

/C 

13 

F+cF 

,cF 

,cF 

,cF 

•C2,3> 

Examp le  2.t:  Here  an  example  is  given  to  show  the  transformation  of  an 
SDTS  into  a  CFPG.  The  transformation  procedure  was  described  in  theorem 

2.3. 

Suppose  that  an  SDTS  is  given  as  follows: 


T  =  (N,  L,  £,R,S) 


where  N  =  CS,A,B,C,D>,  ]T  =  {  a  ,  b  ,  c  ,  d  ,  e  >.  R  contains 

S  *  A BCD,  A BCD,  ABCD 
A  -*■  aA  ,  bA  ,  bA 

A  *  a  ,  b  ,  b 

B  -  dB  ,  eB  ,  eB 

B  *  cB  ,  eB  ,  eB 

B  -*•  d  ,  d  ,  d 

C  ■*  cC  ,  aC  ,  bC 

C  ♦  aC  ,  aC  ,  bC 

C  *  d  ,  a  ,  b 

C  *  dC  ,  iC  ,  bC 

D  *  aD  ,  bD  ,  bD 

D  ♦  a  ,  b  ,  b 
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A  CFPG  G(N '  ,P,S)  is  formed  as  follows: 

The  following  steps  follows  the  procedure  of  theorem  2.3. 


Step  1:  N*  =  -CA^A  e  N>  U  -CA2 ( A  e  N>  U  <A3 ( A  e  UiS> 
=  -CS>  U  CA^B^C^D^S1  |i=1,2,3> 


Step 

2: 

£'  =  £  u 

A  = 

£  U 

£  =  £ 

Step 

3: 

(1)  Add  S 

C1 

S 

s2s3 

l  }  to  P 

.  The  content  of  the  success  fiela 

will 

be 

fi lied  when 

all 

the 

rules  are 

constructed. 

(2)  For  the  rule  in  R:  S  -  A  BCD,  A BCD,  ABCD 
add 


s2  -  a2b2c2d2  i  y 
s3  -  a3b3c3d3  i  > 

to  P 

For  the  rule  in  R:  A  -*•  aA;  bA,  bA 
add 

A1  -  aA1  {  > 

A2  *  bA2  i  y 
A3  -  bA3  i  > 
to  P 

For  the  rule  in  R:  A  ■»  a,b,b 
add 

A^  +  a  -C  y 
A2  -  b  (  y 
A3  -  b  {  ) 


to  p 


and  similarly  for  the  rest  of  the  rules  of  R.  Then  fill  in  those 
success  fields  as  described  in  theorem  2.3.  Then  for  convenience, 
replace  A^B^C^D1,  A2,B2,C2,D2,  A3,B3,C3,D3  by  A,6,C,D,E,F,G,H,W,X,Y,Z 
respectively.  The  following  is  the  resulting  CFPG. 

N'  =  <s,s1,s2,s\ A,B,C,D,E,F,G,H,W,X,Y,Z> 

P  = 


Success 

Success 

Label 

Core 

Field 

Label 

Core 

Field 

1 

12  3 
SjS's'V 

C2> 

17 

X-d 

■Cl  8,1 9> 

2 

S* -♦A  BCD 

C3> 

18 

C-cC 

C22> 

3 

S^-EFGH 

C4> 

19 

C-aC 

C22> 

4 

S+WXYZ 

C5> 

20 

C-dC 

C22> 

5 

A-*-aA 

C7> 

21 

C-d 

C23> 

6 

A  -*a 

C8> 

22 

G-*-aG 

C24> 

7 

E-b£ 

C9> 

23 

G+a 

C25> 

8 

E-b 

C10> 

24 

Y-bY 

C18,19,20,21> 

9 

W+bW 

C5,6> 

25 

Y+b 

C26> 

10 

W+b 

cm 

26 

O+aO 

C28> 

11 

B-dB 

C14> 

27 

D+a 

C29> 

12 

8-cB 

C14> 

28 

H-bH 

C30> 

13 

B+d 

C15> 

29 

H-b 

C31  > 

14 

F-eF 

C16> 

30 

Z+bZ 

C26> 

15 

F-d 

Cl  7> 

31 

Z-b 

$ 

16 

X-eX 

C11,12,13> 

Example  2.7:  The  stochastic 

translation  schema  T(N 

described 

in  Example  2.2 

_» 

is  transformed  to  a  SCFPG 

,P,S)  as  follows: 

Step 

1.  N’  = 

CA1 |A  c  N>  U 

CA2 | A  e  N>  U  CS> 

= 

1111 
CS  ,A  ,8  ,C  , 

2  2  2  2 

S  ,A  ,B  ,C  ,S> 

Step 

2.  £’  = 

T.  u  a  =  co. 

1> 

Step 

3.1.  Put 

the  translation  rules  of  T 

in  the 

order  described  -in 

theorem  2.4,  (They  are  already 

in  order 

in  Example  2.2). 

-kP. 

Step 

3.2.  For 

rule  S  =-±-  0A,  0A 
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Step  3.3. 


Step  3.4. 


Step  3.5. 


Step  3.6. 


add  to  P :  y  S2  *  OA2  S(U>  PCU)  $  1 

2PS 

For  rule  S  —  OA,  1A 

add  to  P:  Y  S2  *  1A2  S(U)  P(U)  $  1 

( y  ,  S(li)  and  P(U)  are  given  later) 

Other  rules  are  added  similarly. 

■pP  -pP 

For  the  rules  S  OA,  OA  and  S  -=~^  OA,  1A 
add  to  P:  y  S1  *  OA1  S(U)  P(U>  $  1 

Other  rules  are.added  similarly. 

For  the  rule  C  -  0,  1 

add  to  P:  y  C2  *  1  <t>  1  <t>  1 

Other  rules  are  added  similarly. 

For  the  rules  C  -=-£•  0,  0  and  C  — -  0,  1 
add  to  P:  Y  C1  *  0  S(U)  P(U)  *  1 
Other  rules  are  added  similarly. 

Add  the  following  rule  to  P: 

1  S  -  sV  S(U)  P(U)  $  1 


The  final  form  of  the  SCFPG  is 
G({S1,A1,B1/C1,S2,A2,B2,C2^S>,-C0,1>,J,P/S) 


P: 

Label 

1 

2 

3 

4 


Core 

U 

P(U) 

s 

+  s^s2 

•C2,3> 

4  ' 

7} 

s1 

-  OA1 

ao,n> 

<Pc' 

Ps> 

s1 

-  1A1 

{10,11} 

<Ps' 

Pc> 

A1 

-  OB1 

t12,13> 

{P  , 

p  > 

c 

s 

A1 

-  IB1 

{12,13> 

<Ps' 

V 

W 

* 


P(W) 

1 

1 

1 

1 


5 


1 
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6 

B1 

oc1 

014,15} 

OP  , 

c' 

V 

1 

7 

B1 

-1 

1C 

014,15} 

/ 

s' 

p } 
c 

<t> 

1 

8 

c1 

0 

016,17} 

OP  , 

c' 

p  } 
s 

<t 

1 

9 

-*■ 

1 

016,17} 

tfv 

V 

1 

10 

s2 

0A2 

04,5} 

4 ' 

4> 

1 

11 

s2 

1  A2 

04,5} 

4' 

iy 

f 

1 

12 

A2 

OB2 

06,7} 

4- 

i> 

f 

1 

13 

A2 

IB2 

06,7} 

i> 

* 

1 

14 

B2 

- 

oc2 

08,9} 

4 ' 

$ 

1 

15 

B2 

-*■ 

1C2 

08,9} 

4' 

<t> 

1 

16 

c2 

-► 

0 

i 

<P 

1 

1? 

c2 

1 

<P 

i 

<t> 

1 

2.6  Conclusion 


The  time-varying  pattern  analysis  problem  is  investigated  in  this 
chapter.  The  problem  is  analyzed  through  the  use  of  deformation  models 
and  translation  schemas.  It  is  shown  that  some  time-varying  patterns 
can  be  characterized  by  a  context-free  programmed  language.  Therefore, 
the  welt  developed  erroi — correcting  string  parser  can  be  applied.  The 
proposed  method  can  be  extended  to  problems  involving  three-dimensional 


patterns. 

Tree  translation 

is  proposed 

to 

analyze  more 

complex 

TV 

patterns. 

Stochastic  translation  is 

also 

presented  to 

mode  l 

the 

stochastic 

properties  of  TV 

patterns . 
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CHAPTER  3 


A  GENERALIZED  SYNTAX-DIRECTED  TREE  TRANSLATION  MODEL 
_3._1_  Introduction 

In  chapter  2,  we  have  proposed  a  syntactic  method  for  time-varying 
pattern  analysis.  An  input  sequence  x^,  x^,  ...,  where  x^  is  the 
pattern  representation  at  time  t^,  is  analyzed  through  a  translation 
model  which  is  defined  as  a  mapping  from  a  language  L^  to  another 
language  in  formal  language  theory.  Consider  L^  as  the  set  of 
possible  patterns  occurring  at  time  and  L^  as  the  set  of  patterns  at 
time  t^.  The  relation  governing  the  time-varying  process  of  the 
sequence  is  then  formulated  as  a  translation  problem.  In  this  chapter, 
the  formulation  of  string  translation  is  extended  to  trees  using  a 
generalized  syntax-di rected  model.  The  generalized  model  is  compared 
with  the  conventional  top-down  and  bottom-up  tree  translation  model.  It 
is  shown  that  both  the  top-down  and  the  bottom-up  models  are  special 
cases  of  the  generalized  model.  A  parsing  algorithm  for  this 
generalized  tree  translation  model  is  presented. 

Definitions  and  notations  that  will  be  referred  to  are  briefly 
reviewed  C57,58,593. 

An  alphabet  i-  is  ranked  by  a  function  r:Z  *  N  which  assigns  a  rank 
to  each  member  of  £.  For  each  n,  I  =  r  (n)  denotes  the  set  of  symbols 
in  £  which  have  rank  n.  Intuitively,  the  rank  of  a  symbol  is  the  number 
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of  sons  it  has  when  it  Labels  a  node  in  a  tree. 

Let  n  denote  the  set  containing  left  and  right  brackets  and  comma. 
To  avoid  possible  confusion,  ranked  alphabets  are  not  allowea  to  include 
elements  of  n.  For  a  ranked  alphabet  X,  the  set  x^  of  (finite  labeled) 
trees  over  the  alphabet  X  is  the  least  set  of  strings  in  (X  (Jii)*  such 
that 

(1)  X  C  X  .  and 

o  -  *' 

(2)  for  n  >  0,  b  e  X^,  and  t^,  t^,  _ ,  tp  e  X*,  bLt^,  t^,  _ , 

t  J  e  X 
n  * 

Definition  3.1 :  If  X  is  a  ranked  alphabet  and  A  is  an  alphabet,  then 
X^(A),  the  set  of  trees  in  X^  indexed  by  A,  is  defined  recursively  as 
follows. 

1)  X  U  a  CZ  X  (a)  . 

o  -  * 

2)  If  b  e  X  n  >  0,  and  t.,...,t  e  X  (A),  then  bLt,,...,t  j 

n  in"  in 

e  X^(A). 

Definition  3.2:  A  (nondeterministic)  top-down  tree  transducer  is  a 
5-tuple  M  =  (Q,  X,  A,  R,  Qq)  where 


(1) 

Q 

is  a 

finite 

set  of 

states. 

(2) 

X 

is  a 

finite 

ranked 

alphabet 

called  the 

input 

alphabet. 

(3) 

A 

is  a 

finite 

ranked 

alphabet 

called  tne 

output 

alphabet. 

(4) 

Q 

0 

c 

Q  is  a 

set  of 

starting 

states,  ana 

(5) 

R 

is  a 

finite 

set  of 

ru  les. 

R  C  U  (Q  x  X  )  x  (A  (J  (Q  x  X  )). 
n>0  n  n 
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A  rule  is  written  in  the  form  (q,b)  *  w,  where  q  e  Q,  b  e  Z  ana  w 

e(6  U  (2  *  x  ))  f°r  some  n,  where  X  denotes  the  set  Lx.,  x-.  ...  x  ) 

for  n  >  0  and  denotes  the  empty  set.  x^  refers  to  the  ith  son  of  the 
current  input  node. 

The  behavior  of  a  top-down  transducer  is  defined  inductively  in 
terms  of  the  output  produced  from  a  tree  starting  in  state  o. 

Definition  ^._3:  Let  M  =  (Q,  z,  &,  R,  Q  )  be  a  top-down  tree  transducer, 
and  let  q  e  Q.  For  a  tree  t  z  Z ,  the  set  of  trees  output  from  t  by  M 
starting  in  state  q  is  denoted  by  M(q,t)  and  is  defined  inductively  as 
f ol  lows . 

(1)  If  t  =  b  e  £^,  then  M(q,t)  =  -Cw|(q,b)  •‘•we  R>; 

(2)  If  t  =  bCt^ ,  t  j  c  l*,  then 

M(q,t)  =  U  w(<p,x  >;M(p,t  • ) |peQ,  1  <  j  <  n) . 

(q,b)+weU  ^  ^ 

Definition  5.4:  A  nondeterminist ic  bottom-up  tree  transducer  L583  is  a 
five-tuple  M  =  (Q,  l.  A,  R,  F)  where 

1)  Q  is  a  finite  set  of  states, 

2)  £  and  A  are  finite  input  and  output  ranked  alphabets, 
respect i ve ly . 

3)  F  C  Q  is  a  set  of  final  or  accepting  states, 

4)  £nn  =  ADn  =  Af1x  =  <i!,  and 

5)  R  is  a  finite  set  of  transition  rules  such  that  every  rule  in  R 
is  either  of  the  form 


( 
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(b)  *  (q,t),  where  b  £  X^,  q  e  Q,  ana  t  e  A^, 
or  of  the  form 

(b,q..,...,q  )  *  (q,t),  where  n  >  0,  b  e  Z  q,  q _ ,q  e  Q,  and  t  e  A  CX  ). 

i  n  n  j  n  *  n 

The  behavior  of  a  bottom-up  transducer  on  an  input  tree  is  defined 
inductively  as  follows. 

Definition  3.5:  Let  M  =  (Q,  Z,  A,  R,  F)  be  a  bottom-up  tree  transducer, 

and  let  q  e  Q.  For  a  tree  t  e  Z  the  set  of  trees  which  M  can  output 

from  t  ending  in  state  q  is  denoted  by  M(q,t)  ana  is  aefinea  inductively 
as  follows: 

(1)  For  b  e  Zq,  M(q,b)  =  -Cw|b  (q,w)  £  R>, 

(2)  For  t  =  bCt1,...,tn]  £  1*, 

M(q,t)  =  tv  | for  some  rule  Cb,q1,. .  .,qn)  (q,w)  £  R, 

some  i,  1  <  i  <  n,  and  some  u.  e  M(q.,t.i,  v  e  Wlx-iu^T 

Definition  3.6:  A  generalized  syntax  directed  translation  (GSDT)  L5/j  is 
a  four-tuple  F  =  (G,  A,  r,  R),  where: 

(1)  G  =  (V,  Z,  P,  S)  is  a  proper  context  free  grammar; 

(2)  A  is  a  finite  set  of  output  symbols; 

(3)  r  is  a  finite  set  of  distinct  translation  symbols  of  the  form 

t.(A),  where  i  is  an  integer  and  A  is  in  V  -  -Cs>,  plus  the  symbol  . 
Whenever  it  is  possible  to  do  so  without  confusion,  we  will  denote  -r^CA) 
by  A..  We  call  A^  the  i-th  translation  symbol  associated  with  A. 

(4)  R  is  a  function  which  associates  with  each  production  A  ■*  a  in 
P,  a  set  of  semantic  rules  (A^  =  0^,  ^  =  B2,...,Am  =  &m>,  in  which  each 
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$ .  is  a  string  in  (r  U  A)  ,  such  that  a  L  L  translation  symbols  uopearing 
in  ? are  translation  symbols  associated  with  nonterminals  appearing  in 

J.  . 

* 

For  each  xml  we  detine  Fix),  the  ^  ”  of  outputs  of  x  as 

follows : 

(1)  If  x  is  net  in  KG),  then  F(x)  =  <5. 

(2)  If  x  is  in  L(G),  then  each  parse  tree  with  yield  x  defines  an 

element  y  in  F(x),  which  is  the  value  of  the  translation  symbol  S. 
associated  with  the  root.  The  value  of  S1  is  computed  bottom-uo  as 

f o l lows  : 

(i)  With  each  interior  node  N  of  the  parse  tree  labeled  A  -  a  are 

associated  the  translation  symbols  A„,  A-,,  ....  A  ,  which  are  ail  the 

1  c  m 

translation  symbols  associated  with  A.  The  values  of  these  translation 
symbols  at  N  are  computed  using  the  semantic  rules  and  the  values  of  the 
translation  symbols  at  the  descendants  of  N  as  follows. 

★ 

(ii)  Suppose  a  is  x  8.x. 8-x-, _ 6,  x;,  where  x.  is  in  l  ana  b  is 

o  1  1  d  2  k  k'  j  j 

in  V,  0  _<  j  <  k .  Suppose  A.  ~  y0c1y1C2y2  ***  C,yl  1  s  the  5emantic  rule 

■Jt 

for  A.,  where  y  is  in  A  and  C.  is  a  translation  symbol  in  r  associated 

i '  j 

with  for  some  1  <  h,  <  k.  Then  vCA.),  the  value  of  A^  at  noce  N,  is 

j  ’  1 

the  string  yQv(C^  )y^v(C2)y2  v(C^)y^  in  4  ,  where  v((K)  is  the  value 

of  C.  at  the  descendant  of  N  which  is  labeled  by  a  production. 

]  '  j 

T(F),  the  translation  defined  by  F,  is  the  set  -C  C  x ,  y )  !  y  c  F(x)>. 

Example  3.1 :  Let  F  =  (G,  {a,b>,  -CS^,  A^,  A->,  8,>,  R),  where  the 

productions  of  the  grammar  and  the  associated  semantic  rules  are: 


Productions 


Semantic  rules 
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(1)  S  A 

(2)  A  -  aAbb 


(3)  A  ♦  oAaB 


(4)  B  -  A 


(5)  A  +  e 


S1  “  A1A2 
A-)  -  aA^B-j 

A^  =  bA-^B^ 
A^  =  aA^B, 

A2  =  DA2B2 
B1  =  A1 
B2  =  A2 
A1  =  e 


A2  =  £ 


F  defines  the  translation  t(w,a1b1)ji  >  0  and  w  e  Ta,b}*,  such  that  w 
has  i  a's  and  i  b's>. 


5.2  Generalized  Syntax-Pi recteo  T ree  Trans  lot  ion 
A  top-down  tree  transducer  (TDTT)  does  not  specify  an  input 

language,  it  assumes  that  all  trees  in  Z are  input  sentences.  A 

bottom-up  tree  transducer  (BUTT)  does  specify  an  input  language,  which 
is  a  subset  of  Z  ,  but  given  an  input  tree  sentence,  even  though  a  BUTT 
can  copy  a  subtree  in  an  output  sentence  many  times,  all  the  copies  of  a 
subtree  are  the  same,  while  a  TDTT  can  generate  different  copies  of  a 
subtree.  There  exist  translations  which  can  be  implemented  by  a  TDTT, 
but  not  by  a  BUTT.  There  also  exist  translations  which  can  be 
implemented  by  a  BUTT,  but  not  by  a  TDTT. 

Example  An  arithmetic  expression  involving  aadition, 

multiplication,  a  constant  c,  and  a  variable  y  may  be  represented  by  a 

tree  over  the  alphabet  Z  =  -C+,  *,  y,  c>,  where  +  and  *  have  rank  2  ana  y 
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and  c  have  rank  0.  We  construct  a  deterministic  top-down  transducer  M 
which  takes  the  formal  derivative  with  respect  to  y  of  the  expression 
represented  by  an  input  tree  in  Z*.  Let  4  =  £  (J  {1,0>  where  1  and  D 
have  rank  0.  Let  M  =  ({D,I>,  Z,  A,  R,  <D>)  be  a  top-down  transducer, 
where  R  contains  the  following  rules: 

<D,  +  )  -*•  +[<D,x1>,<D/x2> 3, 

(D,*)  -  +H*C<D,x1>,<I,x2>3,  *[<I,x1>,<D,x2>33, 

CD,y)  -*•  1,  (D,c)  *  0,  CI,o)  -*•  o  for  o  e{y,c>,  and 

(I,o)  -*  c£<I,x^>,<I,x2>3  for  °  e{+,*>. 

The  tree  translation  defined  by  M  cannot  be  defined  by  a  BUTT.  The 
main  reason  is  due  to  the  second  rule  above.  With  this  rule,  the 
subtree  x^  of  an  input  tree  *Cx^,x23  is  translated  two  times  as  <D,x^> 
and  <I,x1>  respectively.  For  instance. 


+ 


1  c  y  o 


In  the  above  translation,  y  is  translated  two  times  differently  (the 
first  time  as  1  and  the  second  time  as  y).  But  from  the  definition  of 
BUTT,  we  know  that  BUTT  cannot  generate  this  tree  translation  pair.  On 

the  other  hand,  a  translation  which  contains  only  a  single  tree  pair 

*  + 

•C(  /\  ,  /\  >>  cannot  be  implemented  by  a  TDTT  because  a  TDTT  cannot 

y  c  r  o 

specify  finite  input  tree  set.  While  this  translation  can  be 
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implemented  by  the  following  BUTT: 


M=(-Cq0,q1/q2},  {1,0,+>,  R,  {qQ>) 

^ ^q<j  / T )  /  c-*-(q2,Ci) ,  l*,q<j  "*  "*"tx,j  ,x^  j)T. 

Based  on  the  definition  of  6SDT  given  by  Aho  and  Oilman,  we  propose 
a  similar  definition  for  trees,  called  the  generalized  syntax-ch  rectea 
tree  translation  (GSOTT).  It  can  be-  shown  that  both  TDTT  and  BUTT  are 
special  cases  of  GSOTT. 

A  GSOTT  is  defined  as  follows: 

Definition  3.7:  A  generalized  syntax-directed  tree  translation  (GSDTT ) 
is  a  four-tuple  F  =  (G,  A,  r,  R),  where: 

(1)  G  =  (V,  l,  P,  S)  is  a  regular  tree  grammar; 

(2)  a  is  a  finite  set  of  output  symbols; 

(3)  r  is  a  finite  set  of  distinct  translation  symools  of  the  form 
t^(A),  where  i  is  an  integer  and  A  is  in  V  -  IS>,  pLus  the  symbol  . 
whenever  it  is  possible  to  do  so  without  confusion,  we  will  denote  t  .  (A) 
by  A.j .  We  call  A^  the  i  — t  h  translation  symbol  associated  with  A. 

(4)  R  is  a  function  which  associates  with  each  production  A  ■*  a  in 

P,  a  set  of  semantic  rules  {A.  =  A0  =  S->, _ _  A  =  Bm>,  in  which 

each  8-  is  a  tree  in  (r  U^)*,  such  that  all  translation  symbols 
appearing  in  8^  are  translation  symbols  associated  with  nonterminals 
appearing  in  a. 

For  each  x  in  l *  we  define  Fix),  the  set  of  outputs  of  x  as 
follows : 

(1)  If  x  is  not  in  L(G),  then  F(x)  =  <f. 
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(2)  If  x  it  in  L(G),  then  each  parse  tree  with  yielo  x  defines  an 
element  y  in  Fix),  which  is  the  value  of  the  translation  symbol 
associated  with  the  root-  The  value  of  is  computed  bottom-up  as 

follows : 

(i)  With  each  interior  node  N  of  the  parse  tree  labeled  A  -  a  are 
associated  the  translation  symbols  A^,  A^,  ---/  A  ,  which  are  all  tne 
translation  symbols  associated  with  A.  The  values  of  these  translation 
symbols  at  N  are  computed  using  the  semantic  rules  and  the  values  of  the 
translation  symbols  at  the  descendants  of  N  as  follows. 

(ii)  Suppose  -CB ^  j 0  <  j  <  k  for  some  k>  is  the  set  of  nonterminals 
appearing  in  a,  A.  =  is  the  semantic  rule  for  A^  and  TC^|1  <_  j  <  k 
for  some  l>  is  the  set  of  translation  symbols  appearing  in  B_.  with 
being  the  translation  symbol  associated  with  .  Then  v(A),  the  value 

j  1 

of  A^  at  node  N,  is  the  tree  8^  with  each  being  replaced  by  v(C J, 

where  v(C.)  is  the  value  of  C.  at  the  descendant  of  N  which  is  labeled 
J  J 

by  a  Bh  production-  T(F) ,  the  translation  defined  by  F,  is  the  set 

j 

{ (x,y) | y  e  Fix)}. 

Theorem  3.J_:  For  any  BUTT  M  =  (Q,  l,  A,  R,  F)  there  exists  a  GSDTT  F  = 
(G,  A',  T,  R ' )  such  that  M  and  F  define  the  same  translation. 


Proof:  Given  M=(Q,  l.  A,  R,  F)  we  can  construct  F=(G,  A',  r,  R')  as 


follows:  Let  A'  =  A,  and  G  =  (V,  ,  P,  S)  where  l '  =  l .  V=lAq|q  e  Q>, 


S  =  A  ,  q  is  the  accepting  state  and  P  and  R'  are 
qo 

For  each  (b)  -*•  (q,t)  in  R  where  b  e  Z  q  e  Q;  t  e 
t  to  P,  R'  respectively.  For  each  (b,  q^,  ...,  qn) 

e  Z  q.  r  Q  t  e  A  (x.  )  add  A  *  bCA  ,  A  ,  ..., 

n  '  *  h  q  q1  q2 

R'  where  t'  is  the  same  as  t  except  that  each  X.  in 


formed  as  follows: 

A*  add  Aq  *  b'  Aq1  * 

+  (q,t)  where  n>U,  b 


A  D,A  ♦  t '  to  P, 
qn  ql 

t  is  replaced  by 


65 


The  next  thing  is  to  show  that  M  and  F  define  the  same  translation: 

If  <b,t>  e  M,  b  e  Z  ,  t  e  A  ,  it  can  be  found  st rai ghtf orwara  from  the 

construction  procedure  of  F  that  <b,t>  e  F. 

If  <r,t>  e  M,  the  depth  of  r  is  k  and  if  the  statement  that  <r',t*> 

e  M  =>  <r',t’>  e  f  holds  for  all  <r',t'>  where  the  depth  of  r'  is  K-1, 

then  suppose  r=bCr^,  r->,  ...,  rn3  where  b  e  Z^,  r-  e  the  depth  of  r^ 

_<  K-1.  Since  <r,t>  e  M,  there  exists  a  rule  in  R  of  the  form  (b,  q^, 

q^)  •»  (q^,  u)  where  qQ  is  accepting  state  and  u  e  A^(Xn),  besiaes, 

t  is  equal  to  the  tree  of  u  after  each  X.  in  u  being  replaced  by  u^ 

where  u.  e  M(q.,r),  then  there  exists  a  rule  in  R':A  ♦  u  with  each 
111  q01 
x.  of  u  being  replaced  by  A  and  a  corresponding  rule  in  P  :  A  ■+ 
1  q+1  qo 

bCA„  ,  A  ,  . ..,  A  J,  and  A  r.  (*  by  theorem  in  Brainerd  L59j)  the 
q1  q2  qn  qi  1 

depth  of  all  r_.  <  k,  therefore  by  induction  we  know  that  <r,t>  c  M  -> 
<r,t>  e  F.  Similarly,  we  can  show  that  <r,t>  e  F  =>  <r,t>  e  M  therefore 
M  and  F  define  the  same  translation  . 


Theorem  3.2:  For  any  TDTT  M=(Q,  Z,  A,  R,  Q)  there  exists  a  6SDTT  F=(G, 

A',  r,  R')  such  that  M  and  F  define  the  same  translation. 

Proof :  We  prove  it  by  constructing  a  GSDTT  F=(6,  A',  r,  R')  as  follows: 

Let  G=(V,  Z' ,  P,  S),  £'  =  Z,  a'  =  A,  such  that  L(G)  =  (*  the 

existence  of  G  is  proved  in  Brainerd  C59J).  Let  r  =  V  x  Q  i.e.  suppose 

the  number  of  states  in  Q  is  m,  then  A.,  ....  A  c  r  for  A  e  V.  For 

i  m 

each  rule  in  R  :(q,b)  +  w,  where  q  e  Q,  b  e  Z  ,  w  e  A  lu  x  X  )  add  to 

n  *  n 

R':  A^  ♦  u,  where  u  is  the  tree  of  w  after  each  q’X.  being  replaced  by 

B.q,  corresponding  to  each  rule  A  ♦  bCB^  B 2,  ...,  B^]  in  P.  Next  step 

is  to  prove  that  <r,t>  e  M  <==>  <r,t>  e  F:  If  <r,t>  e  M  and  r=b,  b  c  £ 


then  there  exists  a  rule  in  R  :  (qQ,b)  -*■  t  therefore  there  exists  a  rule 

in  P  :  S  -»  b,  and  a  rule  in  R 1 :  +  t  so  <b,t>  e  F.  If  <r,t>  e  M  ana 

r=bCr„,  r_,  ....  r  3,  b  e  t  then  there  exist  a  rule  in  R  :  (q  ,b)  ■*  t' 
i  2  n  n  o 

where  t'  e  A  (Q  x  X  )  and  the  terminal  part  of  t'  can  be  matched  with  t. 
*  n 

Therefore  there  exists  a  rule  in  P  :  S  bllB^,  B^,  ...,  B^J  and  3  rule 

in  R ' :  S  *  u  where  B.  e  V,  u  is  the  result  of  replacing  each  in  t' 
qo  1 

by  B.^.  If  the  statement  that  <r,t>  c  M  =>  <r,t>  e  F  is  true  for  all  r 

of  depth  K-1  then  it's  also  true  for  all  r  of  depth  K.  We  already  know 

that  it's  true  for  K=1  therefore  by  induction  we  know  that  <r,t>  c  M  => 

<r,t>  e  F.  Similarly,  it  can  be  shown  that  <r,t>  e  F  =>  <r,t>  e  M.  So, 

M  and  F  define  the  same  translation. 

Example  ^3.^3:  Let  Z  =  -C+,*,y,c>  be  ranked  alphabet,  where  +  ana  *  have 
rank  of  2,  and  y  and  c  have  rank  0.  We  construct  an  NGSDT  G  which  takes 
the  formal  derivative  with  respect  to  y  of  the  expressions  represented 
by  input  trees  in  £^,  where  c  represents  a  constant.  Let  A  = 
{+, •  ,1  ,0,y,C,3>.  Let  G=(-Cd,I>,£,A,R,{d>),  where 

R=tCd,  +  )-»C(d,x^  )  +  (d,x^)3, 

(d,*)+CC(d,x^)*(I,X2>J+L(I,x1)*(d,x2)J3, 

(d,y)-1,(d,c)*0>  U 

T ( I ,a)*a | ae ty,c>3  (J 

•C(I, a  )-»C(I,x^)a(I,X2)j|ae  <.+,*>>. 

This  TOTT  can  be  replaced  by  a  GSDTT : 


>!•! 


le 
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(d)  k  denotes  the  sequence  number  for  counting  the  sequence  of 
rule  applications. 

(3)  let  X  =  S,  i=1,  <u,v>  =  <(S,  ,  0,  1),  (S^  ,  0,  1)  >. 

(4)  Select  the  first  rule  of  P  for  X.  j=1. 

(5)  replace  node  (X,  ,  l,  k)  of  u  with  rule  X.  in  P.  If  X  is  X  - 

jo  jo 

bCB^,  ...,  then  replace  the  node  with  (X,  b,  j,  i ) L ( x  ,  (j, 

n)/  ***'  (Bm'  '  °'  l):i* 

(6)  Check  compatibility  for  u  and  r,  if  compatible  go  to  (?)  else  go  to 

(11). 

(7)  For  each  node  (Xn,  /  0,  k)  of  v,  where  n  is  any  number,  replace  it 

with  rule  X,  in  R.  If  a  node  in  X.  is  a  terminal  b,  it  is  written  as 

)n  jn 

(X  ✓  b,  j,  i),  if  a  node  in  X-  is  a  nonterminal  B  ,  it  is  written  as 
n  jn  m 

,  0,  i),  go  to  (8). 

(8)  Check  compatibility  for  v  and  t,  if  compatible,  go  to  (9),  else 

(12). 

(9)  Check  if  there  is  any  nonterminal  node  left  in  u,  if  yes,  let  X  be 

the  next  nonterminal,  i=i+1,  go  to  (4),  if  not,  go  to  (10). 

(10)  Check  compatibility  for  u  and  r,  v  and  t,  if  compatible,  find  the 
parse  by  tracing  down  tree  u,  report  success  and  go  to  (12),  if  not,  go 
to  (12). 

(11)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k).  go  to  (13). 

(12)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k),  delete  the  subtrees  being  added  to  v,  restore  their  roots 

with  (Xn,  ,  j,  k)  go  to  (13). 

(13)  check  if  there  are  new  rules  available  for  X,  if  yes,  set  j=j+1. 
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go  to  (5-,  if  not  go  to  (14). 

(14)  If  X=S,  then  finished,  exit.  If  not,  then  backtrack,  take  the 
immediate  father  of  node  (X,  ,  j,  k)  as  new  X,  i=i-1,  and  go  to  (4) 
(Figure  3.1  shows  the  flowchart  for  this  algorithm) 

Example  3.4:  The  translation  rules  of  the  6S0TT  defined  in  example  3.3 
is  labeled,  reordered  and  written  as  follows: 


<sio)  s  *  y 


(sn>  si  +  1 


ts20)  s  -  c 


(S21)  S1  -  0 


(S3q)  S  +  + 


/\ 

A  B 


(s31)  s1  -  + 


A1  B1 


(S4Q>  S 


/\ 

A  B 


(S41}  S1 


/\  /\ 

B-,  A  2 


(A-jq)  A  *  y 


(A^ ^)A.  *-1  (A^ A.,  -*■  y 


(A2Q)  A  -  C 


(A2r  A1  -  0  (A22)  A2  -  C 
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(a30)  A  *  ♦ 


/\ 

A  B 


(A31}  A1  * 


/\ 

A,  B, 


(A32>  a2  -  + 


/\ 

*2  a2 


(A32}  A2  + 


/\ 
A2  B2 


(A40)  A  *  ‘ 


/\ 

A  B 


(A^.|  )  A^  -*■  + 


(aA2)  a2 


A  A 

A1  B2  A2  B1 


/\ 

*2  B2 


(Bio)  b  ■*  y 


(B1 1 )  B1  -  1 


(B12)  B2  -  y 


(b2Q)  B  -  C 


<B21)  B1  -  0 


cb22)  b2  *  C 


(B30)  B  + 


/\ 

A  B 


(B31)  8.,  -  + 


/\ 

A,  B, 


<b32)  b2  -  ♦ 


A 

A2  B2 


(EW  8  *  * 


A 

A  B 


“V  B1  - 


(B42)  82 


•  • 

A  A 

A,  B2  A2  B, 


/\ 

A2  B2 


The  following  diagram  shows  the  analysis  of  the  input  tree  pair 

+  + 

<  /\  ,  /\  >  with  respect  to  this  GSDTT: 
c  y  0  1 


f-goft;  3.1  Flow  chart  for  the  parsing  of  a  6S0TT 
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3.3  Stochasti c  benera  l  i zea  Syntax-Pi rectea  T  ree  T rans  tat  ion 

Def ini t ion  5._8:  A  stochastic  generalized  syntax-directed  tree 

translation  (RGSDTT)  is  a  four-tuple  F  =  (G,  A,  V,  R),  where: 

(1)  C-  -  17,  l,  n,  Si'  is  a  regular  tree  g  -.nisr; 

(2)  &  is  a  finite  set  of  output  symbols; 

(3)  r  is  a  finite  set  of  distinct  translation  symbols  of  the  form 

t  ^  ( A ) ,  where  i  is  an  integer  and  A  is  in  V  -  tS>,  plus  the  symbol  S. . 
whenever  it  is  possible  to  do  so  without  confusion,  we  will  Denote  i-(A) 
by  A^ .  We  call  A.  the  i — t h  translation  symbol  associated  with  A. 

(A)  R  is  a  function  which  associates  witn  a  probability  value  p, 

0  <.  p  <  1  each  production  A  *  a  in  P,  and  a  set  of  semantic  rules  lA^  = 

B. .  A-,  =  B A  -  i  },  in  which  each  6.  is  a  tree  m  (r  U  ^  /  such 

122  mm  l  * 

that  all  translation  symbols  appearing  in  6  are  trans let ’on  sympols 
associated  with  nonterminals  appearing  in  3. 

Definition  3.9:  An  SGSDTT  is: 

(a)  unrestricted  i*  for  each  rule  p:  A  -  a,g  (b  i s  the  set  of  semantic 
rules  associated  with  A  -  i)  the  proDability  o  is  not  conditioned 
on  other  rules  or  even's. 

(b)  proper  if  for  each  nonterminal  A  the  probabilities  o*  all  rules  in 
which  A  is  the  leftside  nonterminal  sum  to  1. 

We  consider  only  unrestricted,  proper  translation. 

Definition  3. 1C:  The  stochastic  translation  from  I  to  A#  produced  by  F 
with  starting  symbol  S  is  the  set 


r 
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n  f  (i ) 

f(F/S)  =  •C(x/y/p)  |  x  in  l*,  y  in  A^,  P  =  £  JJ  j 

i=1  j=1  J 

where  there  are  n  distinct  standard  derivations  (e.g.,  breaath-f i rst)  of 
(x,y)  with  fCi)  steps  in  the  ith  one,  and  P—(x,y)  is  the  probability 
assigned  to  the  jtn  rule  of  the  ith  derivation. 

Simitar  to  the  stochastic  syntax  analysis  for  context-free 
programmed  Languages  CIO],  depending  on  the  type  of  stochastic  selection 
for  the  next  rule,  we  have  two  distinct  types  of  stochastic  syntax 
analyzer  for  tree  translation.  The  first  type  is  one  with  a  selection 
method  which  searches  for  the  most  likely  rule  first,  while  the  secona 
type  is  one  with  a  selection  method  which  randomly  selects  a  rule  for  a 
nonterminal  according  to  the  distribution  over  all  possible  alternatives 
for  the  nonterminal  considered. 

The  stochastic  syntax  analyzer  that  searches  for  the  most  nxely 
rule  first  is  a  nondetermini st i c  syntax  analyzer  in  which  the  rules  are 
arranged  in  descending  order  of  magnitude  of  their  associated 
probabi  lities  C5A □ .  The  first  rule  for  each  nonterminal  is  the  one  witn 
the  highest  probability,  while  the  last  rule  tor  each  nonterminal  is  tne 
one  corresponding  to  the  lowest  probability.  Clearly,  this  is  a 
stochastic  syntax  analyzer  with  a  fixed  strategy  ill)].  The  proieaurt 
for  this  analyzer  is  given  in  Algorithm  3.2.  A  flow  chdrt  is  shown  in 
Fig.  3.2. 

The  stochastic  syntax  analyzer  with  a  random  strategy  is  formed 
from  the  nondeterminist ic  syntax  analyzer  by  incorporating  d  stochastic 
selection  algorithm  in  selecting  the  next  rule  when  alternatives  dre 
available.  At  each  step  the  conditional  probability  distribution  is 


1 

,i 


.1 


f 
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used  in  selecting  the  next  rule  among  all  available  alternatives  i'lQ'j. 
Suppose  that  there  are  3  translation  rules  for  nonterminal  A,  y^,  and 
with  P^,  P^f  Pj  being  their  respective  probabilities,  and  y.j  has  been 
previously  applied.  After  the  backtracking,  y^  and  y-j  are  the  only 
avai lable  rules.  Hence,  either  y^  or  y^  can  be  selected  according  to 

I  I 

their  conditional  probability  distribution  P^2'Y3^Y1^  =  ^P2/P3^  wherp 

I  « 

P^  -  P2^P2+P3^  ant ■*  P3  ~  P3^P2+P3^'  Procedure  f°r  this  analyzer  is 

given  in  Algorithm  3.3  and  a  flow  chart  is  shown  in  Fig.  o.3. 
(Algorithm  3.2  of  the  SGSDTT  parser  with  a  fixed  strategy  is  essentially 
the  same  as  Algorithm  3.1  of  the  nonstochastic  GSDTT  parser.  Algorithm 
3.3  of  the  SGSDTT  parser  with  a  random  strategy  is  almost  the  same  as 
Algorithm  3.1  except  the  rule  selection  method.! 

Algorithm  3.2:  Parsing  for  SGSDTT  with  a  fixea  strategy. 

Input.  An  SGSDTT  F=(G,  A,  T,  R),  G=(V,  £,  P,  S)  and  an  input  tree  pair 
<r,t>. 

Output.  Aa  the  correct  parses  for  <r,t>.  The  output  "error”  it  no 
parse  exists. 

Method . 

(1)  cor  each  nonterminal  A  in  V,  order  the  alternates  in  R  for  A  in 

descending  order  of  magnitude  of  their  associated  probabi  l  i t les .  Let 

A  be  the  index  *or  the  ith  alternate  of  A.  Let  A.  be  the  index  for 
10  1] 

the  ith  alternate  in  R  with  respect  to  A,  in  P. 

10 

(2)  Let  <u,v>  be  a  new  tree  pair.  A  4-tuple  (X,  b,  l,  k)  will  be  useo 
to  denote  the  configuration  of  each  node  in  a  tree: 

(a)  X  denotes  the  nonterminal  of  the  node  before  it  is  rewritten. 
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(b)  b  denotes  the  terminal  of  the  node  after  it  is  rewritten. 

<c)  l  denotes  the  label  of  the  production  rule  for  rewriting  tne 

node. 

Id)  k  denotes  the  sequence  number  for  counting  the  sequence  of 
rule  applications. 

(3)  let  X  =  S,  i=1,  <u,v>  =  <(S,  ,  0,  1),  CS.J,  ,  0,  1)  >. 

(A)  Select  the  first  rule  of  P  for  X.  j=1. 

(5)  replace  noae  (X,  ,  l,  k)  of  u  with  rule  X  in  P.  If  X  is  X 

JO  jo 

bCB.j,  B^,  _ _  Bm3,  then  replace  the  node  with  (X,  b,  j,  i)L(fa.j,  ,  u, 

i),  ...,  (B^,  ,  0,  i)D. 

(6)  Check  compatibility  for  u  and  r,  if  compatible  go  to  (7),  else  go 
to  (11). 

(7)  For  each  node  (X^,  ,  0,  k)  of  v,  where  n  is  any  number,  replace  u 

with  rule  X  in  R.  If  a  node  in  X  is  a  terminal  b,  it  is  written  as 

jn  jn 

(X  ,  b,  i,  i),  if  a  node  in  X.  is  a  nonterminal  fa  ,  it  is  written  as 
n'  '  jn  m' 

(B^,  /  0,  i),  90  to  (8). 

(8)  Check  compat ibi l l ty  tor  e  and  t,  if  compatible,  go  to  (9i  else 

(12). 

(9)  Check  if  there  is  any  nonterminal  nooe  Lett  in  u,  it  yes,  let  X  be 

the  next  nonterminal,  i=i+1,  go  to  (A),  if  not,  go  to  (10). 

(10)  Check  compatibility  for  u  and  r,  v  and  t,  if  compatible,  tino  the 
parse  by  tracing  down  tree  u,  report  success  ano  go  to  (12),  if  not,  go 
to  (12). 

(11)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k).  go  to  (13). 

(12)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 


(X,  ,  j,  k),  delete  the  subtrees  being  added  to  v,  restore  their  roots 

with  (Xn,  ,  j,  k)  go  to  (13). 

(13)  check  if  there  are  new  rules  available  for  X,  if  yes,  set  j=j+1. 
go  to  (5),  if  not  go  to  (14). 

(14)  If  X=S,  then  finished,  exit,  if  not,  then  backtrack,  take  the 
immediate  father  of  node  (X,  ,  j,  k)  as  new  X,  i = i — 1 ,  and  go  to  (4) 
(Figure  3.2  shows  a  flow  chart  for  this  algorithm). 

Algorithm  3.3:  Parsing  for  SGSDTT  with  a  random  strategy. 

Input.  An  SGSDTT  F=(G,  A,  T,  R),  G=(V,  I,  P,  S)  and  an  input  tree  pair 
<r,t>. 

Output.  All  the  correct  parses  for  <r,t>.  The  output  "error"  if  no 
parse  exists. 

Method . 

(1)  For  each  nonterminal  A  in  V,  order  the  alternates  in  R  tor  A.  Let 

A.  be  the  index  for  the  ith  alternate  of  A.  Let  A  De  the  index  for 

10  i  j 

the  jth  alternate  in  R  with  respect  to  A.^  in  P. 

(2)  Let  <u,v>  be  a  new  tree  pair.  A  4-tuDle  (X,  o,  l,  k)  will  Pe  useo 
to  denote  the  configuration  ot  each  node  in  a  tree: 

(a)  X  denotes  the  nonterminal  of  the  node  before  it  is  rewritten. 

(b)  b  denotes  the  terminal  of  the  node  after  u  15  rewritten. 

(c)  l  is  a  register  denoting  which  rules  have  been  applied  tor  the 
node  before. 

(d)  k  denotes  the  sequence  number  for  counting  the  sequence  ot 

rule  applications. 

(3)  let  X  =  S,  i - 1 ,  <u,v>  =  <(S,  ,  0,  1),  (S,,,  ,  0,  1)  >. 

(4)  Select  the  first  rule  of  P  for  X  statistically,  set  j  =  the  index 
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of  the  rule. 

(5)  replace  node  (X,  ,  l,  k)  of  u  with  rule  X.  in  P.  If  X  is  X  ♦ 

jo  jo 

btB^,  ...,  Bm],  then  replace  the  node  with  (X,  b,  l' ,  i)L(b^,  ,  0, 

i),  ...,  (Bm,  ,  0,  i)3,  i,'  =  U  OR  y),  where  y  =  2J  ,  the  jth  bit  of  the 
binary  form  of  y  is  1,  showing  that  the  jth  rule  of  X  is  now  being 
applied. 

(6)  Check  compatiL  lity  for  u  and  r,  if  compatible  go  to  (7)  else  go  to 

(11). 

(7)  For  each  node  (X  ,  ,  0,  k)  of  V,  where  n  is  any  number,  replace  it 

with  rule  X.  in  R.  If  a  node  in  X  is  a  terminal  b,  it  is  written  as 

jn  jn 

(X  ,  b,  l",  i),  if  a  node  in  X.  is  a  nonterminal  B  ,  it  is  written  as 
n  jn  m 

(B^,  >.  0,  i),  go  to  (8). 

(8)  Check  compatibility  for  v  and  t,  if  compatible,  go  to  (9)  else 

(12). 

(9)  Check  if  there  is  any  nonterminal  node  left  in  u,  if  yes,  let  X  De 

the  next  nonterminal,  i=i+1,  go  to  (4),  if  not,  go  to  (ID). 

(10)  Check  compat ibi li ty  for  u  and  r,  v  and  t,  it  compatible,  find  the 
parse  by  tracing  down  tree  u,  report  success  and  go  to  (12),  it  not,  go 
to  (12). 

(11)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k).  go  to  (13). 

(12)  delete  the  subtree  just  being  added  to  u,  restore  its  root  with 

(X,  ,  j,  k),  delete  the  subtrees  being  added  to  v,  restore  their  roots 

with  (Xn,  ,  j,  k)  go  to  (13). 

(13)  check  if  there  are  new  rules  available  for  X,  if  yes,  select  next 
rule  for  X  statistically,  set  j  =  the  index  of  the  rule,  go  to  (3),  if 
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not  go  to  (14). 

(14)  If  X=S ,  then  finished,  exit.  If  not,  then  backtrack,  take  tn 
immediate  father  of  node  (X,  ,  j,  k)  as  new  X,  i = i —1 ,  ana  go  to  (4 
(Figure  3.3  shows  a  flow  chart  for  this  algorithm). 


yes 
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CHAPTER  4 

AN  APPLICATION  OF  TREE  TRANSLATION  TO  TRAFFIC  IMAGE  SEQUENCE  ANALYSIS 

_4._1_  Introduction 

This  chapter  gives  an  illustrative  example  of  applying  the  tree 
translation  model  described  in  Chapter  3  to  the  design  of  an  automatic 
traffic  image  sequence  analysis  system.  Traffic  image  is  a  popular 
subject  of  study  in  time-varying  image  analysis.  The  input  of  the 
analysis  system  is  assumed  to  be  a  sequence  of  images  of  a  traffic 
intersection  scene.  An  example  is  shown  in  Figure  4.1.  In  this  system, 
each  input  -image  is  divided  into  a  set  of  fixed-size,  fixed-position 
windows.  Each  image  is  then  represented  as  a  tree  of  which  each  node 
corresponds  to  a  specific  window  in  the  image.  Each  node  is  labeled  to 
indicate  the  occurrence  or  nonoccurrence  of  a  veh-icle  in  -its 
corresponding  window.  A  tree  translation  scheme  is  then  usea  to 
describe  the  motion  of  the  vehicles  in  the  image  sequences.  Matching  of 
vehicles  in  different  images  is  performed  in  the  form  of  a  tree 
translation  parsing. 


4^_2  Scene  Representation 
4.2.1  Image  representation 

Each  image  is  first  divided  into  a  set  of  windows.  An  appropriate 


window  size  is  selected  so  that  each  window  contains  at  most  one 
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vehicle.  In  general,  the  window  length  selected  should  not  be  greater 
than  the  length  of  the  smallest  vehicle.  Window  size  could  be  set  at  a 
very  small  value.  However,  a  smaller  window  size  will  result  in  a  more 
complex  translation  scheme.  There  are  many  ways  to  divide  the  image. 
An  example  is  shown  in  Figure  4.2.  Only  a  part  of  the  road  area  is 
considered.  A  tree  representation  for  the  image  is  also  shown  in  Figure 
4.2.  Each  window  corresponds  to  a  node  in  the  tree  representation 
regardless  there  is  a  vehicle  in  this  window  or  not.  Each  vehicle  is 
considered  as  only  a  point.  This  point  could  be  the  centroid,  or  any 
easi ly-recogni zable  corner-point  of  the  vehicle.  But  after  this 
reference  point  is  selected,  this  point  should  be  used  consistently  for 
each  vehicle  in  the  whole  image  sequence.  The  centroid  of  a  vehicle 
(considering  only  the  x-y  plane)  is  selected  in  this  system.  A 
vehicle's  position  is  represented  by  the  location  of  its  centroid.  Even 
though  one  large  vehicle  could  occupy  more  than  one  window,  only  the 
window  where  this  vehicle's  centroid  resides  is  considered  as  containing 
this  vehicle.  If  no  vehicle  occurs  in  a  window,  then  the  corresponding 
node  of  this  window  is  labeled  by  'O'.  If  there  is  a  vehicle  in  a 
window,  then  the  corresponding  node  is  labeled  by  the  quantization  value 
of  this  vehicle's  orientation  (see  Figure  4.3).  Additional  information 
(includes  vehicle  size,  actual  centroid  position,  orientation  angle)  is 
attached  to  the  node  for  other  purposes  (e.g.  speed  calculation, 
clearing  of  ambiguity  occurring  in  tree  translation  parsing).  The 
determination  of  a  vehicle's  orientation  and  centroid  position  will  be 
discussed  later.  An  ambiguity  of  tree  translation  parsing  means  that 
different  time-varying  activities  between  images  result  in  the  same 
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Figure  4. 
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2  Divided  intersection  area  and  its  tree  representation. 

(The  root  of  the  tree  is  represented  by  an  encircLed  node.) 
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Figure  4.3  Orientation  primitives 
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sequence  of  trees. 


4.2.2  Motion  representation 

Tree  translation  rules  are  used  to  represent  vehicle  movement. 
Only  the  movement  from  one  window  to  another  is  described.  The  movement 
within  the  same  window  is  not  considered.  But  the  information  about 
vehicle's  centroid  position  and  orientation  is  attached  to  the  node. 
This  information  can  be  used  if  necessary.  It  is  assumed  that  each 
vehicle  can  move  no  more  than  the  length  of  the  smallest  vehicle  between 
two  consecutive  images. 

Example  4.1:  Figure  4.4a  shows  an  intersection  area  being  divided  into  4 
windows.  One  vehicle  is  coming  in  from  the  south  while  another  one  is 
moving  toward  the  south.  The  following  rules  are  required  for  this 
movement: 

S  7  0 

/\  /\ 

A  B  ,  A  B 

A  -  0  „  7  C+0,3 

where  0  means  no  vehicle  existence  and  i  means  the  existence  of  a 
vehicle  in  direction  i,  1  _<  i  <  8. 

Example  _4.j^:  Figure  4.4b  shows  that  one  vehicle  is  moving  from  the  north 
to  the  west  and  two  others  are  moving  from  the  south  to  the  north.  Both 
the  (i-l)th  and  the  ith  images  have  the  same  intersection  content. 
There  are  two  interpretations  for  the  vehicle  moving  west.  Either  the 
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Figure  4.4  Intersection  area  contents,  ("t"  sign  indicates  the 
existence  of  a  vehicle  in  that  direction.) 
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same  vehicle  exists  in  the  same  window  for  both  images  or  the  one  in  the 
(i-l)th  image  has  left  the  intersection  area  and  a  new  one  is  appearing 
in  the  ith  image.  A  similar  interpretation  also  holds  for  those  two 
vehicles  moving  north.  There  can  be  four  different  interpretations  for 
the  variation  between  the  (i-l)th  and  the  ith  images.  But  only  one  set 
of  translation  rules  is  needed  to  describe  this  image  pair. 

S  -  6  6 

/\  /\ 

A  B  ,  A  B 

A  -  0  ,  0 

From  the  assumption  about  vehicle  speed,  this  problem  is  solved  by  using 
vehicle  position  and  orientation  values  attached  to  each  node  of  the 
tree.  For  the  vehicle  with  label  6,  if  the  orientation  value  of  the 
vehicle  in  the  ith  image  is  greater  than  that  of  the  vehicle  in  the 
(i-l)th  image,  then  it  indicates  that  the  vehicle  in  the  (i-l)th  image 
has  left  the  intersection  and  the  vehicle  in  the  ith  image  appears  for 
the  first  time  and  traffic  flow  (the  number  of  passing  vehicles)  is 
increased  by  one.  Otherwise  it  indicates  that  the  vehicles  appearing  in 
the  (i-l)th  and  the  ith  images  are  the  same  ones.  (Orientation  value  is 
counted  counterclockwise)  For  those  vehicles  moving  north,  the  y 
coordinate  of  the  front  vehicle  is  used.  If  the  y  coordinate  of  the 


B  -  3  3 

C  ,  C 
C  -  3  ,  3 


front  vehicle  in  the  ith  image  is  smaller  than  that  of  the  front  vehicle 
in  the  (i-l)th  image,  then  it  indicates  that  the  front  vehicle  in  the 
(i-l)th  image  has  left  the  intersection,  the  front  vehicle  in  the  ith 
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Figure  4.5  Conditions  of  the  (i-l)tb  and  ith  images  for  which  the  use 
of  attribute  value  is  required.  (Each  one  of  (a'  (b)  (c) 
represents  one  of  four  similar  cases.  Blank  windows  could 
contain  vehicle  or  not.  The  attribute  values  of  node 
representing  lower-right  window  will  be  used  for  case  (a) 
and  (b).  For  case  (c)  the  attribute  values  of  the  top- 
right  window  will  be  used.) 
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image  was  the  rear  vehicle  in  the  (i-l)th  image,  the  rear  vehicle  in  the 
ith  image  appears  for  the  first  time  and  the  traffic  flow  is  increased 
by  one.  Otherwise  it  indicates  that  the  front  vehicles  in  both  the  ith 
and  the  (i-l)th  images  are  the  same  one  and  so  are  the  rear  vehicles. 
There  are  also  some  other  similar  cases  which  require  the  use  of 
attribute  values  of  each  node  to  resolve  such  an  ambiguity.  Their 
corresponding  translation  rules  are  specified  for  the  requirement  of  the 
comparison  between  node  attribute  values  of  2  input  trees.  Figure  4.5 
shows  the  various  cases  in  which  a  comparison  of  node  attribute  value  is 
requi red. 


4^  Scene  Analysis 
4.3.1  Feature  extraction 

Due  to  the  similar  property  in  feature  extraction  between  this 
experiment  and  You  and  Fu's  shape  recognition  experiment  C133,  the 
algorithms  for  boundary  following  and  boundary  smoothing  described  in 
C 1 3D  are  applied  for  feature  extraction. 

The  first  step  in  feature  extraction  is  threshold  selection. 
Because  of  the  flat  black  paint  on  the  models,  the  vehicles  look 
uniformly  dark  and  should  create  a  peak  in  the  high  gray  level  region  of 
the  histogram.  The  light  background  is  supposedly  uniform  too  and 
create  a  peak  in  the  low  gray  level  region  of  the  histogram.  A  typical 
histogram  of  the  experiment  is  shown  in  Figure  4.6.  The  peak  at  gray 
level  k^  is  caused  by  the  vehicles.  The  peak  at  gray  level  k^  is  caused 
by  the  background.  The  selected  threshold  is  t. 

Algorithm  Threshold  Selection 
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Input:  A  digital  picture. 

Output:  A  threshold  t. 

Method: 

(1)  Compute  the  histogram. 

(2)  Find  the  peak  of  the  highest  gray  level  (which  usually 
corresponds  to  the  object).  Let  =  the  gray  level. 

(3)  Find  the  second  peak  of  the  histogram  besides  the  one  found  in 
(2).  Let  k^  =  the  corresponding  gray  level. 

(4)  Find  the  lowest  valley  between  the  above  two  peaks  and  let  t  = 
gray  level  corresponding  to  the  valley. 

(5)  Terminate. 

After  a  threshold  is  found,  the  boundary  for  each  vehicle  can  be 

traced  out.  The  boundary  is  defined  as  a  connection  of  edges  between 

the  object  and  the  background.  The  boundary  is  coded  by  unit  vectors 

with  horizontal  and  vertical  directions.  Each  boundary  is  traced  out  by 

the  following  boundary  following  algorithm  which  is  led  by  the  contents 

of  a  2x2  window.  Figure  4.7  shows  the  four  possible  configurations. 

The  pixels  A,  B,  C,  and  0  are  defined  relative  to  the  boundary  vector  P. 

The  object  is  to  the  right  of  P,  so  that  A  is  darker  than  the  threshold 

t.  The  background  is  to  the  left  of  P,  so  that  C  is  lighter  than  t.  In 

the  following  algorithm,  u  and  u  are  the  unit  movements,  or  unit 

x  y 

vectors,  in  the  X  and  Y  directions  respectively.  A,  B,  C,  D  denote  the 
coordinates  of  the  pixels.  6(b)  is  the  gray  level  of  the  pixel 
indicated  by  B.  F  is  the  first  pixel  of  an  object  detected  by  scanning 
the  image. 

Algorithm  4.2:  Boundary  Following  (You  and  Fu  Cl 33 ) 

Input:  F,  u  ,  u  ,  and  threshold  t. 
x  y 

Output:  A  boundary  chain  U  of  i  unit  vectors. 
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Figure  4.6  A  typical  histogram  of  the  experiment 


Figure  4.7  Four  possible  configurations  of  boundary  following  window 
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Method: 

(1)  Set  P  =  u  ,  i  =  1,  11(1 )  =  P 

A  =  F,  C  =  F~u  ,  S  =  C,  go  to  (3) 

(2)  If  (S  =  C)  then  terminate 
otherwise  i  =  i+1,  U(i)  =  P 

(3)  D  =  C+P 

If  (G(D)  <  t)  then  go  to  (4) 
otherwise  P  =  C-A,  A  =  D,  go  to  (2) 

(4)  B  =  A+P 

If  (G(B)  <  t)  then  go  to  (5) 
otherwise  A  =  8,  C  =  D,  go  to  (2) 

(5)  C  =  B,  P  =  B-0  go  to  (2) 

The  output  from  the  above  boundary  following  algorithm  is  a  string 
of  unit  vectors.  Due  to  the  digitization  grid,  there  are  many  zig¬ 
zag's.  A  smoothing  method  (You  and  Fu  Cl 33 )  is  applied  to  transform  a 
string  of  unit  vectors  into  a  string  of  longer  vectors.  This  method  is 
defined  in  the  form  of  an  attributed  finite  transducer. 

In  the  following  definition,  qj's  are  attributed  states.  Each 
state  qj  represents  a  subchain  sj  which  is  accepted  by  not  translated, 
and  which  is  described  by  the  associated  attributes.  iv  denotes  a 
series  of  unit  vector  v,  or  l  times  v.  -v  is  the  negative  of  v,  i.e., 
-v  and  v  have  the  same  length  but  opposite  directions.  A  A  B  denotes 
that  B  follows  A.  6  is  a  mapping  from  Q  x  I,  under  condition  C,  to 
finite  subsets  of  Q  x  0*.  The  mapping  performs  when  condition  C  is 
true.  For  each  state  transition,  there  is  a  set  of  attribute  rules. 
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The  attributes  of  a  state  may  be  unit  vectors  or  numbers.  In  the 
transition  rule,  i,  m,  n,  v,  u  are  unit  vectors  and  p,  q,  t,  k  are 
numbers.  The  attribute  conditions  for  a  transition  are  described  above 
the  right  arrow.  For  each  transition,  the  input  unit  vector  is  compared 
with  the  vector  attributes  and  the  attribute  conditions  are  checked. 
Then  the  machine  goes  to  the  next  state  with  the  appropriate  output  ana 
transfers  the  attributes  according  to  the  attribute  rules.  Each 
expression  of  the  output  is  a  vector  e  0.  An  example  of  the  smoothing 
effect  of  this  transducer  is  shown  in  Figure  4.8. 

Definition  4.1 :  Attributed  Finite  Transducer  A 
A  is  a  6-tuple,  (Q,  I,  0,  6,  S,  F). 

I  =  the  input  set  consisting  of  4  unit  vectors  and  an  end  marker  $, 

{ (1,0), (0,-1 ),(-1,0), (0,1 ),$> 

0  =  the  output  set,  'Cfn^n^)  I  (n^  =  0,n-j_.j  *  0)  or 

(n..  =  ±1,  n^_^  =  any  integer),  i  =  1,2> 

Q  =  a  set  of  states  with  attributes,  fq j | j  =  0,...,9> 

q0:  s0  =  A,  empty 

ql  :  si  has  one  unit  vector  v 

v 

q2t  :  s2  =  tv, — *  ,  *  >  1 

'  tv 

V 

q3v  u :  s3  =  v  A  u,  1  u  (or  J ) 
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Iv 


q4 


l,v,u ' 


s4  =  {.V  A  u. 


(or _ j  ),  i  >  1 


q5 


v,u 


v 

s5  =  v  A  u  A  -v,  u  (or  tD  ) 


q6 


v,u 


s6  =  v  A  u 


A 


_r> 


Q7,  U  ,,  l,:s7  =  A  u  A  kv, 
V /U ✓ * 


(or 


) 


v 

q8.,  „  s8  =  v  A  lu,  i  (or!  *),i  >  1 

v,  ,u  — 5 — * 


Jlv 

q9l,v,uk;  s9  =  £v  A  u  A  kv  A  -u,  - 1  .  1  (or 

kv 

1  >  1,  k  >  1 


S  =  the  initial  state  qO 
F  =  a  set  of  final  states,  -CqO> 


6:  (qO,i)  *  (q1^,$),  v  ♦  i,  i  e  I— C$> 


<q1m'm)  ■"  (q2a  v  ♦  m,  l  *  2 


v  +  m,  u  +  i 
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(q2  ,m)  -  (q2  v  *  m,  i  -  P+1 

p,m  *,v 

(q2  -i)  *  (q4.  w  m,4>),  «•  *  p,  v  -  m,  u  *  i 

p * *  v ,u 

(q3  ,m)  -  (q6  v  *  m,  u  «-  n 

m,n  V/U 

(q3m^n,n)  -  Cq8V/,/U^),  v  -  m/  u  *  n,  i  -  2 

Cq3  ,-m)  +  Cq5  ,<t>),  v  *  m,  u  ♦  n 
m,rr  v,u 

S,  •*■  p,  V  ♦  m 

<q4p,m,n'm)  "  (q7i,v,U, k'*>'  k  +  1,  u  +  n 
<'q4p,m,n'_m)  *  (q3v,u'pm)'  v  ^  n,  u  *  v 

<q4p,m,n'n>  "  (q2*,v'pm)'  4  "  2  V  "  n 
(q5m,n'‘n)  *  <qO'0) 

Cq5  ,-m)  -  <q2,  „,m+n),  i  ♦  2,  v  -  -m 
m,n  «',v 

“»5„,n'n>  *  “  Z'  V  *  " 

v  *■  m,  4  ♦  1 

(q6m,n'~n)  (q9s.,v,u,k'$)/  u  -  n,  k  *  1 
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<p6m  n/n)  *  (q0,m+n  /  m+n) 


(qVn'm) 


u,k/4>)'  v 


l,  k  - 

m,  u  <- 


v  «-  m, 

(q 7  *  (q7  ,  .& ) „ 

Pxm/n,q''  £.,v,u,k'v  u  *  n. 


Cq  7 


p,m,n,q 


,n) 


p>2q 

•*  (ql  ,(p-1)m/  2qm+n),  v  *  n 


q/2<p<2q 

-*■  (ql  ,  (p+q)m+n1 ,  v  ♦  n 


P<q/2 


i  -  q~p 

2pfn+n)'  v  -  m 
u  «-  n 


(q7p,m,n,q'  n) 


^q^s.,v,u,k'  v 


&  +  p 

k  «-  q 

m 

u  «•  n 


(q8  ,n)  ♦  (q8  . 

in,q,n'  v,S,,u 


v  ♦  m 
,  $),  u  *  n 
1  +  q+1 


(q8 


I**  ^q^q 


,m) 


2 

n 

l  *  p 

k  *■  q+1 
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q_<4 


(q4 


l,v,u 


l  «-  q-1 
,m+n)  v  n 
u  *■  m 


q>4 


l  *■  q-2 

<q4J.  v  U'm+2n)  v  ■"  n 
u  *  in 


(q8 


m/,q,n 


,-m) 


q<4 


l  *  q-1 

(q4„  .  m+n)  v  *  n 

Z,v,u' 

u  ♦  -m 


q>4 


1  ♦  q-2 

(q4l,u,v'  m+2n)  V  *  n 

u  +  -m 


<q2l/V,  •  > 


i  *■  m+n+1 
v  ♦  m 


1*2 

(q9p,m,n,q'~n)  +  (q2l,v'  <P+q)m+n>'  y  «.  _n 

v  *  -n 

(q9  „  _  „,-m)  -*■  (q3  -  (p+q)m+n), 

p,m/n,q'  M  v ,u'  K  ^  '  u  ♦  -m 

(q0,$)  ♦  (q0,<t>) 

(q1mr$)  -  (qO,m) 
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(q2p  m,$>  -  (qO, pm) 

<q3m,n'^  “■  (q0'm+n) 

<q4n  m  n'*}  *  (<T°'Pm+n) 

«***,*'»  *  (q0'n) 

(q 6m,n'*)  -  (c<0'2m+n) 

+  (qO,(p+q)m+n) 

(q8m,q,n'$)  *  (q0'm+qn) 

(q9p,m,n/q^)  *  (q0'(p+q)m) 

4.3.2  Shape  analysis 

Consider  the  top  view  first.  An  attributed  shape  grammar  is 

adapted  for  vehicle  shape  representation  C13J.  To  find  the  orientation 

and  the  centroid  position  of  a  vehicle,  special  symbols  are  marked  in 

the  production  rules  of  the  shape  grammar  to  specify  which  pair  of  angle 

points  can  be  used  to  find  the  orientation  and  which  pair  of  angle 

points  can  be  used  to  find  the  centroid  position.  Suppose  that  a 

* 

derivation  of  a  shape  contour  is  S  ♦  F.A,  ...  FA-  _  FA.  ...  A  , 

1  i  11  ]  J  nr 

where  F’s  are  curve  primitives  and  A's  are  angle  primitives,  and  it  is 
known  in  the  grammar  inference  stage  that  the  direction  of  the 


line 
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segment  extending  from  the  point  of  angle  A.  to  the  point  of  angle  A  ^ 

decides  the  orientation  of  this  vehicle  and  it  is  also  known  from 

parsing  that  this  derivation  generates  the  input  boundary  string  ... 

V  _  V  ...  V_  with  substring  V  V  ..  ...  V  matching  F.  .  ...  F.. 

p  q  n  P  P+1  q  1+>  J 

Then  the  position  of  angle  A^  (A ^ )  is  the  same  as  the  position  of  the 
breaking  point  between  vectors  V  „  and  V  (V  and  V  Therefore  the 

orientation  of  this  vehicle  is  decided  with  the  knowledge  of  x  and  y 
coordinates  of  these  two  breaking  points  which  is  obtained  in  the 
feature  extraction  stage.  The  centroid  position  is  calculated 
similarly.  Suppose  that  the  midpoint  of  the  line  segment  connecting  the 
angles  A^  and  A^  is  the  centroid.  Then  the  centroid  position  is  decided 
from  the  x  and  y  coordinates  of  the  angle  points  of  A^  and  A ^ .  You  and 
Fu's  LI  3D  PEE  Earley's  parser  is  used  for  vehicle  shape  recognition. 

For  example,  consider  a  vehicle  shape  shown  in  Figure  4.9.  The 

shape  is  represented  as  S  *  F1A1 F2A2F3A3F4A4*  The  m‘idpo''nt  °'f  the  L'ine 
segment  connecting  the  angles  A^  and  A^  is  the  centroid.  The  direction 
of  the  line  segment  connecting  the  angles  A^  and  A^  determines  the 
orientation  of  the  vehicle.  After  knowing  that  A^  is  located  at 
(x^y^,  A^  is  located  at  (x^^)  and  A^  is  located  at  (x^y^),  then  the 
centroid  position  is  found  to  be  (Cx2+x^)/2,  (y^+y^)/2)  and  the 
orientation  of  the  vehicle  has  the  slope  Cy^-y^ ) / Cx^-x^ ) .  The  question 
about  whether  the  actual  direction  should  be  from  A^  to  A^  or  from  A^  to 
A^  will  be  discussed  shortly.  Strictly  speaking,  for  a  simple  shape 
like  the  top  view  of  a  vehicle,  the  shape  analysis  method  described 
above  is  not  really  necessary.  For  instance,  an  easy  way  to  calculate 
the  centroid  (x  ,y  )  of  a  vehicle  is  to  apply  the  formula:  x  *  /n, 


y  =£y/n,  where  the  summation  is  taken  over  all  points  on  the  boundary 
c  i 

and  n  is  the  total  points  on  the  boundary.  The  orientation  can  also  be 
easily  determined  after  searching  for  the  right  angles  around  the 
boundary  and  finding  out  the  longest  line  segment  between  these  angles. 
But  compared  with  this  simpLe  method,  the  method  described  earlier  in 
this  subsection  does  serve  as  a  way  for  the  determination  of  the 
centroid  and  orientation  of  a  general  shape. 

Due  to  the  symmetrical  property  of  the  top  view  of  a  vehicle,  a 
vehicle  in  any  direction  could  have  two  possible  opposite  orientations. 
For  example,  the  shape  of  a  vehicle  moving  northwest  could  also  be 
evaluated  as  southeast  -  oriented.  This  problem  is  solved  as  follows: 
Consider  a  vehicle  located  in  the  lower-right  window  of  the  intersection 
area.  If  its  shape  is  detected  as  horizontally  oriented  (either  to  the 
east  or  west),  then  it  is  considered  as  moving  east  with  orientation  1. 
Similarly,  if  its  shape  is  detected  as  vertically  oriented,  then  it  is 
considered  as  moving  north  with  orientation  3.  If  its  shape  is 
northwest  -  oriented  (in  other  terms,  southeast  -  oriented),  it  could 
have  orientation  A  or  8.  this  ambiguity  is  resolved  from  the  knowledge 
of  the  last  image  and  the  assumption  about  vehicle  movement  limitation 
between  consecutive  images.  Specifically,  if  there  exists  a  vehicle 
with  orientation  8  in  either  the  lower-left  or  the  lower-right  window  of 
the  intersection  area  in  the  (i-l)tb  image,  then  this  northwest-oriented 
vehicle  in  the  lower-right  window  of  the  ith  image  should  have 
orientation  8  instead  of  A.  Otherwise  this  vehicle  should  have 
orientation  A.  For  a  northeast-oriented  vehicle  in  the  lower-right 


window,  there  is  only  one  choice: 


orientation  2. 


The  way  of 
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determining  the  orientation  of  vehicles  in  the  lower-right  window 
described  above  also  applies  to  other  windows. 

When  an  input  image  does  not  represent  the  top  view,  the  vehicle 
shape  variation  and  occlusion  problems  happen.  For  shape  variation  and 
simplp  occlusion,  the  problem  can  still  be  solved  with  the  help  of  a 
generalized  error-correcting  PEE  parser  (6ECPEEP)  Cl 30 .  But  when  the 
number  of  vehicles  appearing  in  the  "scene  increases,  a  serious  occlusion 
would  make  the  extraction  of  vehicle  centroid  position  and  orientation 
information  very  difficult.  Therefore,  in  order  to  make  the  monitor 
working  even  under  heavy  traffic  conditions,  it  would  be  better  to  use 
the  top-view  of  the  scene  as  the  input. 

A. 3. 3  Tree  translation  parsing 

After  the  tree  representation  for  the  ith  image  is  obtained,  it  is 
parsed  by  the  tree  translation  parser  along  with  the  tree  representation 
for  the  (i-l)th  image.  (Parsing  process  starts  from  the  second  image.) 
The  translation  parser  is  similar  to  the  one  shown  in  Chapter  3.  But  as 
mentioned  before,  some  conf igurations  of  input  tree  pair  still  require 
the  use  of  attributes  attached  to  the  node  to  decide  the  actual  activity 
between  two  consecutive  images.  After  the  parsing,  the  parser  either 
reports  the  output  or  asks  for  a  comparison  of  attribute  values.  (The 
conditions  requiring  the  use  of  attributes  and  which  attributes  to  be 
used  have  been  discussed  earlier  and  illustrated  in  Figure  4.5.) 

Inference  of  a  Structure-Preserved  T ree  Translation  Schema 

The  tree  translation  involved  here  is  structure-preserved  (in  other 
words,  substitution  transformation  only).  The  inference  of  the  tree 
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translation  schema  is  therefore  essentially  a  tree  grammar  inference 

problem:  Let  £  be  the  input  alphabet,  A  be  the  output  alphabet  of  a  tree 

translation  and  S+  =  •C<x^,y^>/  <x2'y2>"“^  be  the  tree  translation 

sample  set.  x..  and  y^  have  the  same  tree  structure,  form  a  new 

alphabet  £  =  Ex  A.  Transform  each  tree  pair  <x-,y.>  into  a  single 

tree  over  J2  by  combining  the  labet  of  each  node  of  x^  and  the  label  of 

its  corresponding  node  of  y...  The  tree  translation  sample  set  S+  is  now 

transformed  into  a  set  of  tree  samples.  Apply  the  conventional  tree 

grammar  inference  procedure  (Moayer  and  Fu  C61 □ >  to  infer  the  tree 

grammar  which  can  generate  these  tree  samples.  Then  transform  this  tree 

grammar  into  the  required  tree  translation  schema  by  converting  each 

ab 

production  rule  into  a  tree  translation  rule.  Ce.g.  if  A  +  X--\  is  a 

B1  bn 

rule  of  the  tree  grammar,  where  a  e  X.  and  b  e  a,  then  the  converted  tree 

a  b 

translation  rule  is  A  ♦  /.. \  ,  /.  .\  >.  The  whole  procedure  is 


given  as  follows: 


81  Bn  B1  Bn 


Algorithm  4.3:  Inference  of  a  structured-preserved  tree  translation 
schema 

Input:  A  tree  translation  sample  set  S+  =  •C<x^,y^>,  *2'y2>* •  •<xn,yn», 
where  x.  and  y.  have  the  same  tree  structure  (tree  domain), 
input  alphabet  £  and  output  alphabet  A. 

Output:  a  tree  translation  schema  which  generates  S+. 

Method:  1.  Let  £  =  5E  x  A ,  S++  =  $. 

2.  For  each  <x.,y ,>  €  S+  , 
add  a  tree  (a:  D  *  23  '  to  S++, 
where  0  is  the  domain  of  x.  and  y., 
a(b)  =  ax^(b)ay.(b)  for  b  e  0, 
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a  .  and  a  .  are  the  tree  function  for  x.  and  y.  respectively. 

*  '  /  '  1  1 

3.  Infer  a  tree  grammar  for  S++  (Moayer  and  Fu  C61 □ ) : 

Step  1:  Selection  of  proper  substructures,  here  the  aim  is  to 
select  components  which  have  high-repetitive 
occurrence . 

Step  2:  Infer  subgrammar  6.  for  each  substructure 

Step  3:  Let  G^  be  the  union  of  the  subgrammars  G.  inferred  in 
step  2.  Relabel  the  variables  and  eliminate  redundant 
production  rules  vo  eliminate  ambiguity  and  erroneous 
generation  paths. 

4.  Form  a  tree  translation  schema  T(  £,a,y,R,S)  by  adding  one 

a  b  ab 

rule  A  /.  ,\  ,  X.\  to  R  for  each  rule  A  -*•  X--\  of  the 


B.  B  B.  B 
1  n,  .1  ,  n 
grammar  generated  in  3. 


81  Bn 


Consider  the  tree  translation  involved  in  the  traffic  scene  as  an 
example.  The  input  alphabet  and  output  alphabet  are 
£  =  <0,1 ,2, 3, 4, 5,6,7, 8>.  To  reduce  the  number  of  samples, 
consider  only  the  image  pairs  in  which  no  vehicle  exists  in  the 
intersection  area  of  the  first  image  and  one  vehicle  comes  into  the 
intersection  area  of  the  second  image.  The  tree  translation  sample  set 
contains: 


0— G  0—0 


0  0  0  2 


N  ,  < 


0—0  0—0 


0  0  0  3 
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0—0  0—0  0—0  0— A 

<  I  1,1  I  >  ,  <  I  1,1  !>, 

0004  0000 


0—0  0—5  0—0  0—6 

<  I  1,1  I  >  ,  <  I  1,1  I  >  , 
0000  0000 


0—0  6—0  0— '0  7—0 

<  I  1,1  l>,<  I  1,1  I  >  , 

0000  0003 


0—0  8—0  0—0  0—0 

<  I  1,1  l>,<  I  i,l  l>, 

0000  0080 


0—0  0—0  0—0  o—o 

<  I  I  ,  I  I  > ,  <  I  I  ,  I  I  > . 

0  0  1  0  0  0  2  0 


These  tree  translation  samples  are  transformed  into  trees  over  1]  *  i. 


Then  a 

tree  grammar 

is 

generated 

from  these  tree 

samples.  For  example. 

0—0 

0—0 

00—00 

<  1  1 

,  1  1  > 

becomes 

1  I- 

The  tree  sample 

set  contains: 

0  0 

0  2 

00  02 

00—00 

00—00 

00—00 

1  1  r 

1  1, 

1 

1 

✓ 

00  02 

00  03 

00 

04 

00—04 

00—05 

00- 

•06 

1  1  / 

1  1  / 

1 

1 

/ 

00  00 

00  00 

00 

00 

Ill 
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1 

00 

00 

B1  *  I 

B2  +  1 

1 

h 

A , 

C„ 

4 

1 

1 

00 

04 

1 

®3  *  1 

C2 

B‘*i 

A1 

! 

05 

06 

< 

B5  *  ] 

B6  +  1 

A1 

A1 

00 

1 

A1 

A1  -  00 

Az  -  08 

* 

a3  -  01 

A4  -  02 

\  ■ 

C1  -  03 

C2  *  04 

where  S,  A^, 

■••'A4'  B1 r ■ " ■ 

,B-?,  and  Cz 

are  nonterminals. 

The  resulting  tree 

translation 

schema  contains  the  following 

translation 

rules: 

0 

0 

0 

0 

S  -  /\ 

,  /\ 

s  -  /\  , 

/\ 

A1  B1 

A1  B1 

A1  B2 

Ai  B2  ; 

t 

A 

J 

0  0 


A1  A1 


A1+0,0  A^  -  0  ,  8 
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A3-0,1  A4-0r2 


0  ,  3  C2+0/4 

Since  there  are  4  windows  in  the  intersection  area,  there  are  4 

nodes  in  each  tree  and  8  nodes  in  each  translation  pair.  There  are  9 

different  possible  labels  for  each  node:  {0,1  ,2, . . .  ,83.  Theoretically, 
4  8 

there  are  9  different  trees  and  9  different  translation  pairs.  But 

0—0  0—0 

after  deleting  some  unrealistic  cases  (e.g.,  a  tree  like  |  |  or  |  |  is 

0—0  0—0  0  5  0  7 

not  reasonable,  a  translation  pair  like  <  |  |  ,  |  |  >  is  not  possible 

0  3  0  0 

due  to  the  assumption  about  speed  limitation),  there  are  25  different 
tree  patterns  and  186  different  translation  pairs  left.  These  186 

translation  pairs  are  divided  into  2  groups.  No  vehicle  moves  out  of 
the  intersection  area  for  each  translation  pair  of  group  I,  whiLe  one  or 
more  vehicles  have  moved  out  of  the  intersection  area  for  each 

translation  pair  of  group  II.  Group  I  consists  of  70  members.  The  12 
translation  pairs  given  in  the  above  inference  example  are  part  of  these 
70  pairs.  Group  II  consists  of  116  members.  These  116  translation 
pairs  are  further  classified  into  12  subclasses  according  to  the 
specific  traffic  type.  These  12  subclasses  are  EW,  ES,  EN,  WE,  WS,  WN, 
SE,  SW,  SN,  NE,  NW  and  NS,  where  E,  W,  S  and  N  denotes  east,  west,  south 
and  north  respectively  and  EW  represents  the  subclass  of  translation 
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pairs  for  which  one  vehicle  has  moved  from  the  east  to  the  west  ana  left 

the  intersection  area,  ES  represents  the  subclass  of  translation  pairs 

for  which  one  vehicle  has  moved  from  the  east  to  the  south,  etc.  For 
0—3  0—0  0—4  0—0 

example,  <  |  I  ,  |  I  >  belongs  to  the  subclass  SN  and  <  |  I  ,  I  I  > 
0000  0000 

belongs  to  the  subclass  EN.  There  are  some  translation  pairs  which 

0—3  0—0 

belong  to  more  than  one  subclass.  For  example,  <  |  |  ,  |  |  >  belongs 

7  0  0  0 


to  both  subclasses  SN  and  NS. 


_4._5  Implementation 

The  whole  experiment  of  traffic  image  sequence  analysis  was  set  up 
in  the  laboratory  of  pattern  processing  and  advanced  automation  at 
Purdue  Oniversity.  The  set-up  for  data  collection  is  shown  in  Figure 
4.1.b.  A  white  background  with  black  vehicles  is  used  for  the  traffic 
scene.  Road  areas  and  non-road  area  were  specified.  The  digital 
picture  sequence  was  taken  directly  through  a  TV  scanner  which  is 
located  above  the  traffic  intersection.  The  digitization  process  was 
controlled  interactively  through  a  POP  11/45  computer  in  the  Laboratory. 
Before  digitization,  we  adjusteo  the  relative  distance  and  the  focus  of 
the  TV  scanner  to  obtain  a  reasonably  clear  picture  on  a  TV  monitor. 
Then  everything  kept  stationary  except  the  vehicles  during  the 
digitization  of  the  whole  image  sequence.  After  each  digitization  of  a 
picture,  the  vehicles  were  moved  to  new  locations  for  the  next  picture. 

The  traffic  analysis  system  using  the  proposed  tree  translation 
parsing  is  implemented  in  Fortran  IV  under  Unix  system  on  a  PDP11/45 
computer.  The  average  time  required  for  the  analysis  of  one  image  is 
about  15  seconds  (the  actual  computer  time  for  each  individual  image 
depends  on  the  number  of  vehicles  in  the  image).  About  90%  of  the  time 


Figure  4.10  Flow  chart  for  the  analysis  of  each  image  (A  flow  chart 
tree  translation  parsing  is  given  in  Figure  3.1.) 
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is  spent  on  the  feature  extraction  and  the  shape  analysis  for  tree  node 
labeling. 

Figure  4.11  shows  the  image  contents  of  some  of  the  image  sequence. 
Figure  4.12  shows  their  corresponding  tree  representations  around  the 

intersection  area.  The  correspondence  between  the  nodes  and  the 

windowed  regions  is  shown  in  Figure  4.2.  The  8  nodes  corresponding  to 
the  windows  surrounding  the  intersection  area  are  included  to  indicate 
where  the  vehicles  come  from  and  where  they  go.  Figure  4.13  shows  the 
traffic  information  extracted  from  the  image  sequence.  This  experiment 
illustrates  an  example  of  the  application  of  tree  translation  models  to 
the  analysis  of  time-varying  patterns.  In  chapter  2,  we  let  denote 

the  set  of  pattern  representations  of  some  object  occurring  at  time  t.^, 
\~2  denote  the  set  of  pattern  represenations  occurring  at  time  t^  and  use 

the  concept  of  translation  to  model  the  relation  between  and  L-,  Cor 

the  time-varying  process  of  the  pattern  representation  between  time  t^ 
and  time  t^).  In  this  traffic  scene  experiment,  the  "object"  under 
study  is  the  content  of  the  intersection  region  in  a  traffic  scene. 
and  are  a  set  of  trees  a:  D  Z,  where  D  is  the  tree  domain 

<0,0.1 ,0.2, 0.2. 1>  and  l  is  the  primitive  set  <0,1, 2, 3,. . .,8>.  Actually 

and  are  the  same  set.  As  mentioned  in  section  4.4,  (or  L2)  has 
25  patterns  and  there  are  186  possible  translation  pairs  between  and 
<2.  These  186  translation  pairs  are  divided  into  2  groups.  Group  1 
involves  no  traffic  flow  out  of  the  intersection  area.  Group  II 
involves  traffic  flow  and  is  further  classified  into  12  subclasses.  At 
each  time  instant  T=t^,  the  tree  representations  of  the  (i-l)th  and  i-th 
images  are  parsed  to  decide  the  membership.  If  they  belong  to  group  I, 
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then  there  is  no  traffic  flow  out  of  the  intersection  area  and  no 

information  update  is  required.  If  they  belong  to  group  II,  then  an 

action  for  the  specific  subclass  is  taken:  update  the  traffic 

information  for  the  specific  path.  For  example,  the  tree 

representations  of  the  intersection  area  for  image  SC01  and  image  SC02 

0—5  5—0 

of  Figure  4.11  are  |  |  and  |  j  which  are  generated  by  the  following 

0  0  10 

translation  rules: 


0  5  5  0 

SI  -  /  \  ,  /  \  B12  -  |  ,  | 

A3  B12  A3  B12  C3  C3 


A3  -0,1  C3  -  0  ,  0 


They  belong  to  group  I.  Therefore  no  action  is  taken  and  the  traffic 

information  is  unchanged.  At  the  next  time  instant,  the  tree 

5—0 

representations  for  image  SC02  and  image  SC13  of  Figure  4.10  are  |  | 

0—0  1  0 
and  |  |  ,  which  are  generated  by  the  following  translation  rules: 

0  1 

5  0  0  0 

S2  -  /  \  ,  /  \  B17  -  j  ,  | 

A12  B17  A12  B17  C13  C13 


A12  -1,0  C 1 3  -0,1 


They  belong  to  the  subclass  EW  of  group  II.  Therefore  the  traffic 
information  is  updated  and  the  number  of  vehicles  moving  from  the  east 
to  the  west  is  incremented  by  one.  (In  case  that  a  translation  pair 
belongs  to  two  subclasses  as  mentioned  in  section  4.4,  then  it  is 
required  to  update  the  traffic  information  of  the  paths  specified  for 
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both  subclasses.)  The  image  sequence  of  this  experiment  does  not  have 
any  significance  of  physical  meaning.  It  is  only  used  to  show  how  the 
analysis  system  is  applied.  (Since  there  are  186  possible  translation 
pairs  (in  other  words,  186  time-varying  patterns  between  consecutive 
images),  in  order  to  go  through  all  of  the  time-varying  patterns,  the 
required  number  of  images  is  far  greater  than  186.  It  is  similar  to  the 
input  sequence  required  to  test  every  state  transition  of  a  finite-state 
machine.)  This  experiment  shows  a  form  of  pattern  recognition  system 
under  time-varying  situation.  It  also  demonstrates  one  way  to  extract 
information  from  an  image  sequence  through  a  series  of  pattern 
translation  operations.  On  the  other  hand,  although  the  testing  image 
sequence  is  successfully  analyzed,  further  improvement  of  the  system  is 
still  required.  Similar  to  the  cases  of  Chow  and  Aggarwal  C43D  and 
Aggarwal  and  Duda  C42D,  this  system  assumes  a  fairly  simple  image 
condition  for  preprocessing.  Therefore,  to  make  the  system  working 
under  noisy  conditions,  more  sophisticated  feature  extraction  technique 
will  certainly  be  required. 

4^6>  Conclusions  and  Discussions 

While  many  traffic  scene-related  research  activities  123,68, 69, 80J 
concentrate  on  segmentation  techniques,  the  proposed  traffic  analysis 
system  emphasizes  the  representation  of  vehicle  motion  and  assumes  that 
there  is  little  difficulty  in  segmentation,  which  is  the  main  assumption 
of  the  system.  The  advantages  of  the  proposed  system  include:  (1)  each 
moving  object  (vehicle)  is  allowed  to  have  movement  ranging  from  0  to 
the  length  of  the  smallest  vehicle  between  consecutive  images,  (2)  the 
matching  process  is  performed  through  a  tree  translation  parsing  which 
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is  very  efficient  in  processing,  (3)  there  is  no  need  to  do 
sophisticated  prediction  using  information  of  the  past  history,  (A) 


there  is  no 

need 

to 

keep 

the  past 

information  except  tne  tree 

representation 

of 

the 

last 

image,  and 

(5)  the  ability  to  describe  an 

image  scene  and  to  model  an  image  sequence.  The  experiment  conducted 
here  is  a  2-lane  traffic.  When  the  number  of  traffic  lanes  increases, 
the  required  analysis  system  is  essentially  the  same  except  that  more 
windows  are  required  and  the  corresponding  tree  representation  contains 
more  nodes.  Figure  A.1A  shows  a  4-tane  example.  There  are  16  windows 
in  the  intersection  area.  The  scene  is  represented  as  a  tree  with  16 


nodes 
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SC01 


SC05 


Figure  4.11  Image  contents  of  the  first  8  images  in  tht 
sequence 


testing  image 
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Figure  4.12  Tree  representations  for  the  images  of 
Figure  4.11. 
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I.N.  EW  ES  EN  WE  WS  WN  SE  SW  SN  NE  NW  NS 
2000000000000 
3100000000000 
4100100000000 
5100100000000 
6100100000000 
7200100000000 
8200100000000 
9200200000000 
10  200200000000 
11  200200000000 
12  200200000000 

13  200200000000 

14  200200000000 

15  200200000000 

16  200200000000 

17  200200000000 

18  200200000000 

19  201200000000 

20  201200000000 
21  201200000000 
22  201200000000 

23  201  200000000 

24  201200010000 

I.N. - image  number 

EW - no.  of  vehicles  going  from  east  to  west 

ES - no.  of  vehicles  going  from  east  to  south 

EN - no.  of  vehicles  going  from  east  to  north 

WE - no.  of  vehicles  going  from  west  to  east 

WS - no.  of  vehicles  going  from  west  to  south 

WN - no.  of  vehicles  going  from  west  to  north 

SE - no.  of  vehicles  going  from  south  to  east 

SW - no.  of  vehicles  going  from  south  to  west 

SN - no.  of  vehicles  going  from  south  to  north 

NE - no.  of  vehicles  going  from  north  to  east 

NW - no.  of  vehicles  going  from  north  to  west 

N'S - no.  of  vehicles  going  from  north  to  south 


Figure  4.13  Traffic  information  extracted  from 
the  image  sequence 
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Figure  4.14  A  4-Lane  intersection  area  being  divided  into  16  windows 
and  its  tree  representation  (with  the  encircled  node  as  the 
root) 
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CHAPTER  5 

CONCLUSIONS  AND  SUGGESTIONS  FOR  FURTHER  RESEARCH 
_5.1_  Cone  lusions 

A  syntactic  method  for  the  analysis  of  time-varying  image  patterns 
is  proposed  and  studied.  This  method  utilizes  a  translation  schema  to 
model  time-varying  properties.  A  syntactic  deformation  model  is  first 
applied  to  transform  the  i-th  image  into  the  (i+1)-th  image  of  an  image 
sequence.  Then  the  concept  of  translation  in  formal  language  theory  is 
used  as  a  mechanism  to  characterize  the  dynamic  process  of  the  sequence. 
In  order  to  analyze  high-dimensional  patterns,  the  string  translation 
schema  is  extended  to  that  for  trees.  Formulation  of  stochastic 
translation  is  also  presented  for  the  modeling  of  stochastic  properties 
of  time-varying  patterns. 

A  traffic  scene  analysis  problem  is  analyzed  using  the  proposed 
method.  Each  input  image  is  presented  as  a  tree  structure.  Tree 
translation  is  used  to  model  the  variation  of  image  content  between 
consecutive  images.  A  parsing  algorithm  for  the  tree  translation  is 
applied  to  match  vehicles  in  each  pair  of  consecutive  images.  The 
advantages  of  this  system  are:  (1)  Each  moving  object  is  allowed  to  have 
larger  flexible  movement  between  consecutive  images,  (2)  the  matching 
process  is  performed  through  a  tree  translation  parsing  which  is  very 
efficient  in  processing,  (3)  there  is  no  need  to  do  sophisticated 
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prediction  using  information  of  the  past  history,  (4)  there  is  no  need 
to  store  the  past  information  except  the  Last  image,  and  (5)  has  the 
potential  for  simulation  of  the  time-varying  process.  This  system 
assumes  little  difficulty  in  the  segmentation  of  moving  object,  which  is 
its  main  limitation. 


5.2  Suggestions  for  Further  Research 

The  problem  of  time-varying  pattern  analysis  is  a  new  research 
topic  and  is  getting  increasing  attention.  Although  the  results  from 
the  proposed  syntactic  method  appear  to  be  quite  satisfactory,  there  are 
still  some  topics  requiring  further  investigations. 

(1)  The  translation  models  studied  in  this  report  include  string 
translation  and  tree  translation  models.  There  are  some  other  patterns 
which  are  better  being  represented  as  graphs  CIO, 96, 98, 99] .  Recently 
Jacobus,  Chien  and  Selander  analyze  the  similarities  in  the  structure  of 
abstract  forms  produced  from  stereoscopic  motion  picture  sequences  using 
a  graph  matching  technique  C72J.  A  pair  of  images  contains  the  scenes 
of  a  3-D  object  before  and  after  the  movement.  The  informtion 
concerning  the  boundary  and  surface  of  the  object  at  each  image  is 
represented  as  a  graph.  A  graph  matching  method  is  used  to  match  the 
corresponding  parts  of  the  two  graphs  (assuming  small  overall  movement). 
In  order  to  make  graph  representation  applicable  for  general  time- 
varying  patterns,  it  would  be  desirable  to  study  graph  language 
translation  and  stochastic  graph  translation. 

(2)  You  and  Fu  C133  study  the  syntactic  shape  recognition  using 
attributed  grammars  with  promising  results.  Tsai  and  Fu  [14,963  report 
an  attributed  pattern  deformation  model  and  find  that  sometimes  it  is 
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more  powerful  tc  use  attributed  pr  mitives  than  discrete  primitives.  In 
order  to  take  attributes  into  consideration  in  time-varying  pattern 
problems,  one  way  is  to  extend  the  translation  models  to  translation 
models  with  attributes:  attributed  string  translation,  attributed  tree 
translation  and  attributed  graph  translation. 

(3)  Tree  translation  is  applied  to  analyze  traffic  image  sequence 
in  Chapter  4.  In  order  to  further  understand  the  usefulness  of  tree 
translation,  it  would  be  desirable  to  study  traffic  scenes  using  tree 
translation  through  an  extensive  simulation  experiment. 

(4)  The  problem  of  cardiac  motion  analysis  is  getting  more 
attentions  191,92,93,94,95].  The  use  of  computerized  tomography 
technology  is  also  becoming  more  popular.  It  would  be  interesting  to 
study  the  applicability  of  the  translation  models  described  in  this 
research  to  the  problem  of  heart  wall  motion  representation,  or  more 
generally,  to  the  problem  of  representation  of  data  extracted  from 
computerized  tomography. 
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A  syntactic  method  for  the  analysis  of  time-varying  image  patterns  is  proposed 
I  and  studied.  This  method  utilizes  translation  schema  to  model  the  time-varying 
properties  of  image  patterns.  A  syntactic  deformation  model  is  first  applied  to  trans- 
I  form  the  i-th  image  into  the  (i+?)-th  image  of  an  image  sequence.  Then  the  concept 
i  of  translation  in  formal  language  theory  is  used  as  a  mechanism  to  characterize  the 
dynamic  process  of  the  image  sequence.  A  formulation  of  stochastic  translation  is  also 
presented.  A  generalized  syntax-directed  tree  translation  model  is  proposed  to  handle 
high-dimensional  patterns.  The  generalized  model  is  compared  with  the-  conventional 
top-down  and  bottom-up  tree  translation  models. 

A  traffic  monitoring  problem  is  analyzed  using  the  proposed  tree  translation 
model.  Each  input  image  is  represented  as  a  tree  structure.  The  proposed  tree 
translation  model  is  used  to  model  the  variation  of  image  content  between  consecutive 


f  7.  K  ■  i '.  *  ’ '  n.)  .  .  n  1 7a.  I  r  i ;  ■ 

images.  A  parsing  algorithm  for  tree  translation  is  applied  to  match  moving  objects 
(vehicles)  in  each  pair  of  consecutive  images. 
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